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I.  INTRODUCTION 


A.  PURPOSE  /  PROBLEM  STATEMENT 

Millions  of  dollars  are  spent  each  year  on  quick  reaction  capability  (QRC) 
Infonnation  Warfare  (IW)  and  intelligence  collection  systems.  These  infonnation 
technology  (IT)  systems  are  used  to  fill  existing  capability  gaps  at  the  Fleet,  theater  and 
strategic  level.  Managing  these  portfolios  of  equipment  as  an  investment  requires  that 
program  managers  be  able  to  maximize  the  benefits  of  the  systems  in  a  systematic 
manner  while  considering  the  impact  of  cost  throughout  the  enterprise.  To  successfully 
accomplish  this  task,  it  is  essential  to  develop  a  means  and  methodology  to  measure  the 
performance  of  the  IT  investments  in  an  objective  manner.  As  of  yet,  no  acceptable 
measure  of  their  performance  or  benefit  has  been  developed  to  determine  whether  the 
systems  are  yielding  an  adequate  level  of  perfonnance  or  return  on  investment  (ROI). 

B.  BACKGROUND 

Intelligence,  as  stated  in  the  National  Security  Strategy  (NSS)  of  the  United 
States,  is  the  nation’s  first  line  of  defense  against  terrorists  and  other  hostile  nations.  The 
document  calls  for  innovation  within  the  armed  forces  to  transform  intelligence  and 
exploit  the  country’s  advantages  in  science  and  technology,  stating  the  following: 

Innovation  within  the  anned  forces  will  rest  on  experimentation  with  new 
approaches  to  warfare,  strengthening  joint  operations,  exploiting  U.S. 
intelligence  advantages,  and  taking  full  advantage  of  science  and 
technology  [...]  We  must  transform  our  intelligence  capabilities  and  build 
new  ones  to  keep  pace  with  the  nature  of  these  threats./ 

According  to  the  NSS,  intelligence  capabilities  must  be  transformed  and  new  capabilities 
must  be  built  to  keep  pace  with  emerging  threats.  Intelligence  systems  must  be 
appropriately  integrated  with  defense  and  law  enforcement  systems  as  well  as  Allied 
systems.  The  NSS  calls  for  hard  choices  to  “ensure  the  right  level  and  allocation  of 
government  spending  on  national  security.”2  It  also  calls  for  an  investment  in  future 
intelligence  capabilities  and  a  strong  security  infrastructure  to  prevent  the  compromise  of 

1  United  States.  President.  The  National  Security  Strategy  of  the  United  States  of  America. 
Washington.  White  House.  2002.  p.  30.  <http://www.whitehouse.gov/nsc/nss.pdT>  [1  June,  2005]. 

2  Ibid.  pp.  30-31. 
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those  capabilities.  Lastly,  the  NSS  acknowledges  the  managerial  changes  that  must  be 
made  stating  that  “[w]e  must  also  transfonn  the  way  the  Department  of  Defense  is  run, 
especially  in  financial  management.”3 

Transformation  of  the  business  practices  internal  to  the  Department  of  Defense 
(DoD),  to  include  financial  and  program  management,  is  a  key  enabler  for  achieving  the 
intelligence  goals  of  the  NSS.4  With  this  mandate,  the  DoD  established  the  Defense 
Acquisition  System,  DoD  Directive  5000.1,  dated  12  May  2003,  to  manage  the 
technological  investments,  resources  and  programs  required  to  accomplish  the  NSS  and 
support  the  Armed  Forces.  The  primary  objective  of  the  DoD  5000  Series  is  to  acquire 
“quality  products  that  satisfy  user  needs  with  measurable  improvements  to  mission 
capability  and  operational  support,  in  a  timely  manner,  and  at  a  fair  and  reasonable 
price.”  5  To  realign  the  individual  services’  intelligence  efforts  with  the  DoD 
transformational  practices,  coordinative  mechanisms  such  as  the  Joint  Intelligence 
Surveillance  and  Reconnaissance  (ISR)  Operational  Concept  and  the  Joint  ISR 
Operational  Architecture  (ISR  JOC/JOA),  have  been  created.6 

1.  Navy  ISR 

The  Naval  Transformation  Roadmap  of  2003  calls  for  the  reengineering  of 
maritime  ISR  to  align  with  the  DoD’s  5000  Series  and  joint  warfighting  concepts  with  the 
goal  of  redefining  standards  and  metrics  and  ensuring  interoperability,  while  providing 
the  warfighter  required  capabilities  in  a  timely,  cost-effective  and  efficient  manner. 7 
Maritime  ISR  lies  at  the  core  of  the  Naval  Operational  Doctrine  and  is  an  essential 
element  in  improving  the  speed  and  effectiveness  of  naval  and  joint  operations.  It  is 
necessary,  with  today’s  threat,  to  expand  the  range  of  ISR  options  available  to  the 
commander  and  ensure  decision  superiority  across  the  range  of  military  operations  in 
accordance  with  the  NSS.  From  a  program  management  point  of  view,  there  are  several 

3  United  States.  President.  The  National  Security  Strategy  of  the  United  States  of  America. 
Washington.  White  House.  2002.  p.  30.  <http://www.whitehouse.gov/nsc/nss.pdT>  [1  June,  2005]. 

4  Ibid.  31. 

5  Department  of  Defense.  DoD  Directive  5000.1.  “The  Defense  Acquisition  System.”  12  May  2003.  p. 

2. 

6  Department  of  the  Navy.  Naval  Transformation  Roadmap  2003:  Assured  Access  &  Power 
Projection. .  .From  the  Sea.  Washington:  Dept,  of  the  Navy,  2003.  pp. 68-69. 

7  Ibid.  69. 
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challenges  in  meeting  future  requirements.  First,  program  mangers  must  create  a 
requirements  management  program  that  will  thoroughly  capture  the  joint  warfighters’ 
seemingly  intangible  inputs  and  translate  them  as  accurately  as  possible  to  ensure  the 
system  will  satisfy  the  user  requirements.  Next,  needs  and  requirements  must  be 
transformed  into  a  comprehensive  set  of  product  and  process  system  descriptions  to 
“solve”  the  ISR  problem  in  a  top-down  manner  by  applying  the  Systems  Engineering 
Process  (SEP).  The  requirement  for  the  integration  of  cutting-edge  technology  and 
limited  resources  must  be  leveraged  with  the  use  of  Commercial  Off-The  Shelf  and 
Government  Off-The-Shelf  (COTS/GOTS)  systems  in  an  Open  Architecture  (OA)  that 
will  integrate  seamlessly  with  existing  systems.  Lastly,  innovative  techniques  to 
metricize  Key  Performance  Parameters  (KPPs)  must  be  developed  to  facilitate  integrated 
Testing  and  Evaluation  of  Navy  and  Joint  ISR  programs. 

With  this  backdrop,  the  goal  of  Navy  ISR  program  managers  is  to  develop  an 
investment  strategy  for  maritime  ISR  systems  consistent  with  validated  warfighter 
requirements  and  within  the  framework  of  approved  architectures.  The  strategy  should 
adhere  to  a  national-level  strategic  view,  and  balance  the  development  of  new 
intelligence  collection  capabilities,  modernization  of  current  systems,  and  sustainment  of 
existing  infrastructures,  as  well  as  move  toward  robust  interoperability.  The  DoD  5000 
Series  and  the  Joint  Capabilities  Integration  and  Development  System  (JCIDS), 
delineated  in  the  Chairman  Joint  Chiefs  of  Staff  Instruction  (CJCSI)  3 170.0  IE,  dated  11 
May  2003,  and  other  governing  documents  such  as  the  Government  Performance  and 
Results  Act  (GPRA),  and  the  Information  Technology  Management  and  Results  Act 
(ITMRA)  serve  as  a  guidelines  for  addressing  these  challenges. 

2.  The  Cryptologic  Carry-On  Program 

The  Cryptologic  Carry-On  Program  (CCOP)  is  a  product  of  the  Advanced 
Cryptologic  Systems  Engineering  program,  which  develops  state-of-the-art  Intelligence, 
Surveillance  and  Reconnaissance  (ISR)  capabilities  in  response  to  Combatant  Command 
requirements  for  a  quick-reaction  surface,  subsurface  and  airborne  cryptologic  carry-on 
capability.  CCOP  systems  have  broad  scope  and  functions.  There  are  approximately  100 
cryptologic  capable  surface  ships  in  the  current  Navy  inventory.  Each  of  these  ships  is  a 
potential  user  of  carry-on  equipment.  In  addition,  there  are  numerous  subsurface  and  air 
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platforms  that  are  also  potential  users.  CCOP  provides  the  necessary  resources  to  enable 
a  rapid  transition  of  available  Commercial  off  the  Shelf  (COTS)  and  Government  off  the 
Shelf  (GOTS)  technologies  that  apply  ISR  system  functionalities.  These  technologies 
typically  require  various  levels  of  integration  to  leverage  on-board  capabilities  that 
provide  system  and  mission  management,  product  reporting  and  data  analysis  support. 
COTS/GOTS  systems  usually  require  some  level  of  adaptation  or  modification  to  meet 
fleet  requirements.  Before  deployment  for  operational  use,  systems  must  be 
systematically  tested  to  ensure  suitable  and  reliable  operation.  They  must  also  be  tested 
for  network  vulnerabilities  (if  connected  to  Navy  or  Joint  networks),  and  tested  against 
joint  interoperability  requirements.  8 

3.  Acquisitions  Implications 

The  Department  of  Defense  Directive  5000.1,  12  May  2003,  states  that  advanced 
technology  shall  be  integrated  into  producible  systems  and  deployed  in  the  shortest  time 
practicable  to  address  “capability  gaps”  between  stated  national  security  threats  and  the 
ability  to  properly  address  them  with  current  capabilities.9  Evolutionary  Acquisition 
(EA)  is  the  preferred  DoD  strategy  for  rapid  acquisition  of  mature  technologies.  An 
evolutionary  approach  delivers  capability  in  increments,  recognizing,  up  front,  the  need 
for  future  capability  improvements.10  The  CCOP  shares  this  objective  of  EA,  which  is  to 
balance  needs  and  available  capabilities  with  resources,  and  to  put  capability  into  the 
hands  of  the  user  quickly. 

a.  COTS/GOTS  and  Open  Architectures 

The  DoDD  5000.1  also  states  “A  modular,  open  systems  approach 
(MOSA)  shall  be  employed,  where  feasible.”  Historically,  cutting-edge  ISR  systems 
were  costly  and  often  “stovepiped.”  They  were  not  readily  adaptable  or  reconfigurable 
for  a  rapidly  changing  global  threat.  The  DoD  mandated  use  of  MOSA  and  a  migration 
to  open  systems  architectures  that  incorporate  commercial  solutions  to  address  the  fast- 

8  Department  of  Defense.  “The  Advanced  Cryptologic  Systems  Engineering  Program:  Budget  Item 
Justification.”  Washington:  February,2004. 

<http://www.dtic.mi1/descriptivesum/Y2005/Navy/0204574N.pdf#search-Advanced%20Cryptologic%20S 
ystems%20Engineering%20program'>  [April  2005] 

9  DoDD  5000.1,  12  May  2003.  p.  2. 

10  Department  of  Defense.  DoD  Instruction  5000.2.  “Operation  of  the  Defense  Acquisition  System.” 
12  May  2003.  p.  3. 
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paced  threat.  A  number  of  information  systems  architectures,  policies,  procedures  and 
standards  have  been  developed  and  implemented  to  enable  the  interoperability  of  these 
new  systems  and  facilitate  the  migration,  11  MOSA  enables  CCOP  systems  to  be 
designed  for  affordable  change  with  reduced  life  cycle  costs.  It  also  facilitates  the 
employment  of  evolutionary  acquisition  with  more  opportunities  for  technology  insertion 
and  shortens  the  total  acquisition  time. 

b.  Risk  Management 

The  Risk  Management  Guide  for  DoD  Acquisition  (June  2003)  defines 
risk  management  as  “the  art  and  science  of  panning,  assessing,  and  handling  future  events 
to  ensure  favorable  outcomes.  The  alternative  to  risk  management  is  crisis  management.” 
The  use  of  COTS/GOTS  and  MOSA  in  the  CCOP  all  provide  a  level  of  risk,  which  must 
be  managed.  The  increasing  system  technological  complexities,  the  reliance  on  complex 
software,  along  with  shortened  acquisitions  timelines  and  funding  instability  all 
contribute  to  this  risk.  Modeling  and  simulation  (M&S)  tools  should  be  used  to  mitigate 
some  of  the  risk  in  the  CCOP.  In  particular,  cost  models  should  be  employed  to 
determine  projected  lifecycle  costs  along  with  manpower  and  performance  M&S  tools. 
The  Defense  acquisition  Guide  Book  (Nov  2004)  suggests  that  alternatives  to  the 
traditional  cost  estimation  need  to  be  considered  because  legacy  cost  models  tend  not  to 
adequately  address  costs  associated  with  infonnation  systems.12 
C.  RESEARCH  OBJECTIVES 

The  objective  of  this  research  is  to  develop  a  decision  support  model  and 
methodology  to  assist  in  the  budgeting  process  for  the  United  States  Navy’s  Chief  of 
Naval  Operations  (OPNAV)  CCOP  Program  Office  (OPNAV  N201)  acquisition  of 
information  warfare  systems.  The  model  will  assess  the  effectiveness  and  efficiency  of 
the  CCOP  portfolio  of  IW  systems  in  support  of  the  achievement  of  the  Navy  ISR 
mission.  The  methodology  will  be  used  to  develop  quantitative  objectives  that  are  based 
on  measurable  data  and  quantifiable  criteria.  Once  measures  are  defined,  an  objective 
performance  baseline  for  each  system  in  the  portfolio  can  be  created  with  which  to 

11  Department  of  Defense.  Maritime  SIGtNT  Architecture  Technical  Standards  Handbook  v.  1.0.  Ft. 

George  G.  Meade,  MD:  NSA/CSS,  23  August  1999.  p.  6.  ~  ~  . . 

12  Department  of  Defense.  The  Defense  Acquisition  Guidebook.  The  Defense  Acquisition  University. 
Sect.  4. 5. 7.2 
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compare  the  impact  of  existing  and  future  CCOP  systems  on  individual  activities, 
processes  and  operations.  The  resulting  infonnation  can  then  be  utilized  to  make  sound 
financial  decisions  and  projections  in  the  acquisition  and  deployment  of  these  systems. 

D.  METHODOLOGY 

This  thesis  will  attempt  to  model  the  intelligence  collection  process  (ICP),  as  it 
applies  to  shipbome  signals  intelligence  (SIGINT)  collection,  and  measure  the  impact  of 
CCOP  Systems  on  the  process  utilizing  the  Knowledge  Value-Added  (KVA) 
methodology.  First,  all  assets,  subprocesses,  and  outputs  will  be  identified.  The  analysis 
of  assets  will  encompass  all  cost  data  related  to  each  asset  in  the  process,  human  and  IT. 
The  subprocess  analysis  will  contain  a  detailed  breakdown  of  the  ICP  to  include  the  time- 
to-leam  and  number  of  executions  for  each  subprocess.  The  process  outputs  will  also  be 
analyzed  to  include  the  total  number  of  process  outputs  and  a  surrogate  revenue  stream 
that  will  be  used  to  monetize  the  outputs.  Next,  asset  costs  will  be  allocated  throughout 
the  subprocesses  in  which  they  contribute  to  the  production  of  outputs.  The  time-to- 
learn,  or  knowledge  embedded  in  each  subprocess  will  be  multiplied  by  the  number  of 
executions  of  that  subprocess.  The  resulting  figure  will  be  used  as  a  basis  for  the  KVA 
Approach  for  allocating  revenue  at  the  subprocess  level.  Once  a  cost  and  benefit  for  each 
subprocess  has  been  detennined,  performance  ratios  such  as  ROA  and  ROI  can  be 
calculated. 

Comparing  performance  measures  of  various  configurations  of  CCOP  systems 
within  the  shipborne  ICP  can  assist  program  managers  in  identifying  targets  of 
transfonnation,  which  will  aid  in  making  higher-level  acquisition  strategy  decisions. 
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II.  RETURN  ON  INVESTMENT 


A.  WHAT  IS  ROI? 

While  there  are  many  forms  of  “return”  such  as  return  on  equity  (ROE),  return  on 
performance  (ROP)  and  return  on  assets  (ROA),  the  focus  of  this  research  is  the 
overarching  concept  of  Return  on  Investment  (ROI). 

1.  ROI  Defined 

Return  on  Investment  or  ROI  has  been  defined  in  many  ways,  but  it  has  generally 
been  known  to  encompass  the  quantification  of  the  past,  present,  and  potential  future 
performance  of  an  organization  or  business.  More  specifically,  the  return  on  investment 
in  assets  is  a  measure  of  perfonnance  that  is  detennined  by  the  percentage  relationship  of 
earnings  to  assets.13  Many  companies  and  organizations  view  ROI  both  as  a  goal  as  well 
as  a  measure  of  profit  and  asset  perfonnance.  In  its  most  generic  fonn,  it  is  characterized 
by  the  following  equation:^ 

Earnings 

rercentageROI  - - 

Investment 

In  this  equation,  earnings,  or  the  net  operating  income  of  an  organization  or  business  is 
the  difference  between  revenue  and  expenses.  Investment  can  also  carry  a  wide 
connotation,  but  in  a  general  case,  the  amount  of  investment  is  detennined  by  an 
examination  of  an  organization’s  original  capital  and  its  assets,  both  financial  and 
physical.  Thus,  a  higher  return,  or  percentage  ROI,  indicates  a  more  efficient  and 
effective  use  of  those  assets  and  capital. 

Another  view  of  ROI,  by  Clarence  Nickerson,  a  Professor  of  Business 
Administration  Emeritus  at  the  Harvard  University  Graduate  School  of  Business 
Administration,  recognizes  that  “the  value  of  a  business  property  is  dependent  on  what  it 


13  Nickerson,  Clarence  B.  Accounting  Handbook  for  Nonaccountants.  3rd  Ed.  New  York:  Van 
Nostrand  Reinhold  Co.,  1986.  p.  73. 

!4  Ibid.  632. 


7 


can  produce.”15  He  also  states,  “in  order  to  judge  the  value  of  the  wealth  created,  we 
should  take  into  account  the  property  required  to  produce  it.”i6 

As  a  practical  mater,  ROI  can  be  used  to  detennine  the  potential  benefit  of  a 
single  investment  strategy  or  of  multiple  strategic  investment  options.  Forecasting 
revenue  streams  and  comparing  them  with  the  expected  capital  investment  and  operating 
costs  of  a  single  strategy  over  the  course  of  time,  or  of  multiple  strategies  can  accomplish 
this.  ROI  yields  valuable  insight  for  managers  and  investors  making  high-level  business 
strategy  decisions.  But  what  if  an  organization  produces  no  measurable  “revenue,”  in  the 
business  sense  of  the  word? 

B.  ROI  IN  THE  PUBLIC  SECTOR 

As  the  budgets  of  public  sector  organizations  are  coming  under  increased 
scrutiny,  stakeholders,  managers  and  taxpayers  are  all  demanding  a  higher  level  of 
accountability  and  transparency  of  the  public  investment  in  those  organizations. 
Additionally,  regulations  such  as  the  Government  Perfonnance  Results  Act  of  1993 
(GPRA),  require  the  establishment  of  strategic  planning  and  performance  measurement  in 
programs  for  the  accountability  of  their  expenditures.  Faced  with  these  challenges, 
public  sector  organizations  have  been,  for  many  years,  searching  for  a  defensible  means 
of  producing  the  metrics  required  to  measure  the  value  of  the  services  and  products  they 
provide.  In  some  cases,  an  organization  may  produce  outputs  that  are  similar  to  those  of 
a  commercial  organization  or  business,  allowing  that  organization  to  directly  place  a 
value  to  its  products  and  services  based  on  the  market  price  or  value  of  the  commercial 
products.  But  many  public  sector  organizations  are  inherently  different,  and  require  a 
different  approach  to  the  determination  of  their  ROI. 

1.  The  Public  Sector  Problem 

There  are  several  issues  that  must  be  considered  when  conducting  an  ROI  analysis 
in  the  public  sector.  First,  the  frequent  absence  of  measurable  revenues  and  profits  make 
it  a  challenge  to  detennine  the  overall  benefit  stream  produced  by  the  organization. 
Secondly,  hard  data  is  often  difficult  to  collect  amidst  an  abundance  of  seemingly 

15  Nickerson,  Clarence  B.  Accounting  Handbook  for  Nonaccountants.  3rd  Ed.  New  York:  Van 
Nostrand  Reinhold  Co.,  1986.  p.  652. 

!6  Ibid. 
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intangible,  soft  data.  Also,  ROI  depends  on  the  notion  of  costs  and  benefits.  In  the 
public  sector  though,  the  recipients  of  the  benefits  or  stakeholders  are  not  easily 
identifiable;  potential  beneficiaries  could  be  program  participants,  managers  of 
participants,  sponsors  of  the  program  or  even  taxpayers  who  are  concerned  about  their 
tax-dollars.  Another  problem  lies  at  the  very  nature  of  public-sector  organizations  and 
government  services  in  particular,  whose  services  are  “essential”  and  must  be  provided, 
regardless  of  the  accountability  or  cost.  Lastly,  public-sector  organizations  do  not  often 
have  the  range  of  flexibility  or  options  to  correct  problems  and  make  radical  changes.17 

2.  The  Public  Sector  Reality 

The  reality  of  public-sector  organizations  is  that  they  do  provide  benefits, 
although  in  most  cases  these  are  presented  in  the  fonn  of  cost  savings  resulting  from 
improved  quality,  increased  productivity  or  timesaving  initiatives.  Monetizing  the 
benefits  directly  may  be  difficult,  but  new  approaches  to  the  problem  are  currently  being 
developed.  The  issue  of  hard  data  is  also  one  that  can  be  overcome;  output,  quality,  cost 
and  time  are  all  usually  measurable  in  private  as  well  as  public-sector  organizations  and 
represent  viable  categories  of  hard  data.  The  notion  that  services  provided  by 
government  organizations  are  essential  and  above  scrutiny  is  also  being  dispelled; 
regulations  such  as  the  GPRA  are  requiring  all  agencies  to  undergo  evaluation  to  ensure 
they  are  operating  effectively  and  efficiently.18  And  finally,  identifying  flexibility  or 
options  in  public-sector  organizations  is  not  only  possible,  but  required  if  a  desire  exists 
to  improve  the  effectiveness  of  public  programs  and  services. 

This  research  focuses  on  public  sector  ROI  as  it  applies  to  the  Department  of 
Defense  and  a  portfolio  of  information  technology  (IT)  assets.  Because  of  the  variable 
nature  of  IT  and  their  benefits,  determining  ROI  on  them  poses  some  unique  challenges. 

C.  IT  ROI 

Organizations,  public  and  private,  make  investments  in  information  technology 
with  the  goal  of  improving  the  efficiency,  timeliness,  quality  and  effectiveness  of  their 

17  Phillips,  Patricia  P.  “ROI  in  the  Public  Sector.”  Ed.  Patricia  Phillips.  Alexandria,  VA:  American 
Society  for  Training  and  Development,  2002.  <http://www.astd.org/NR/rdonlyres/779B0547-EBlF-48E5- 
87 7 6-7808 1 07FD0 1 0/ 0/Whitepaper ,pdf#search='ROI%20in%20the%20public%20sector'>  [August  2005] 

18  Ibid. 
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services,  products  or  processes.  The  problem  these  organizations  face  though,  is 
discerning  the  amount  of  value  contributed  by  their  IT  investments  on  the  processes  in 
which  they  are  meant  to  operate. 

1.  Current  Approaches 

There  have  been  numerous  approaches  to  assessing  the  impact  of  IT  on  an 
organization’s  fiscal  performance  at  the  corporate  and  sub-corporate  levels  of 
aggregation.  Table  1  summarizes  some  of  the  most  common  approaches  to  measuring 
the  return  on  IT  and  the  advantages  and  limitations  of  each. 


Level  of 
Analysis 

Approach 

Focus 

Example 

Key 

Assumption 

Key 

Advantage 

Limitation 

Process  of 
Elimination 

Treats  effect  of 
IT  on  ROI  as  a 
residual  after 
accounting  for 
other  capital 
investments 

Knowledge 

capital 

ROI  difficult 

to  measure 
directly 

Uses 

commonly 

accepted 

financial 

analysis 

techniques 

and  existing 

accounting 

data 

Cannot  drill  down  to 
effects  of  specific  IT 
initiatives 

Aggregate 
Corporate 
(firm)  level 

Production 

Theory 

Determines  the 
effects  of  IT 
through  input 
output  analysis 
using  regression 
modeling 
techniques 

Economic 
production 
function 
links  IT 
investment 
input  to 
productivity 
output 

Uses 

econometric 
analysis  on 
large  data 
sets  to  show 
contributions 
of  IT  at  firm 
level 

"Black-box"  approach 
with  no  intermediate 
mapping  of  IT's 
contributions  to  outputs 

Resource- 

Based 

View 

Linking  firm 
core 

capabilities 

with 

competitiveness 

Uniqueness 
of  IT 

resource  = 

competitive 

advantage 

Strategic 
advantage 
approach  to 
IT  impacts 

Causal  mapping  between 
IT  investment  and  firm 
competitive  advantage 
difficult  to  establish 

Corporate/ 

sub- 

eorporate 

Option 

Pricing 

Model 

Determines  the 
best  point  at 
which  to 
exercise  an 
option  to  invest 
in  IT 

Timing 

exercise 
option  = 
value 

Predicting 
the  future 
value  of  an 

IT 

investment 

No  surrogate  for  revenue 
at  sub-  corporate  level 
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Family  of 
Measures 

Measure 
multiple 
indicators  to 
derive  unique 
contributions  of 
IT  at  sub¬ 
corporate  level 

Balanced 

score-card 

Need 
multiple 
indicators  to 

measure 

performance 

Captures 
complexity 
of  corporate 
performance 

No  common  unit  of 
analysis/  theoretical 
framework 

Sub¬ 

corporate 

(Process) 

Cost-Based 

Use  cost  to 
determine  value 
of  information 
technology 

Activity- 

based 

costing 

Derivations 
of  cost  ~ 
value 

Captures 
accurate  cost 
of  IT  ! 

No  surrogate  for  revenue 
at  sub-  corporate  level  — 
no  ratio  analysis 

Knowledge 

Value 

Added 

Allocating 
revenue  to  IT  in 
proportion  to 
contributions  to 
process  outputs 

Housel  & 
Kanevksy 
KVA 

Methodology 

IT 

contributions 
to  output  ~ 

IT  value- 
added 

Allocates 
revenue  and 
cost  of  IT 
allowing 
ratio  analysis 
of  IT  value- 
added 

Does  not  apply  directly 
to  highly  creative 
processes 

Table  1.  Common  Approaches  to  Measuring  Return  on  IT.  19 


These  approaches  are  subdivided  into  two  distinct  levels  of  analysis,  the  corporate  level 
and  the  sub-corporate  level.  Corporate-level  approaches  seek  to  determine  the 
contribution  of  the  assets,  both  knowledge  and  IT,  on  the  overall  performance  of  the 
organization.  The  sub-corporate-level  approaches  look  internally  at  the  sub-processes 
which  are  involved  in  the  production  of  organizational  output  and  attempt  to  establish  a 
measure  for  the  benefits  of  knowledge  and  IT  assets  within  each  sub-process.  The  goal 
of  each  approach  is  to  provide  managers  with  a  metric  of  value  for  their  organization’s 
investment  in  IT  and  knowledge  assets. 

D.  MANAGING  IT  INVESTMENTS  IN  THE  DOD 

It  is  the  stated  goal  of  the  Department  of  Defense  (DoD)  that  all  infonnation 
technology  (IT)  programs  within  the  organization  be  managed  as  investments  rather  than 
acquisitions.  To  accomplish  this,  and  to  fall  within  the  framework  of  regulations  and 
legislation  such  as  the  Government  Perfonnance  and  Results  Act  (GPRA)  and  the 
Information  Technology  Management  and  Results  Act  (ITMRA),  the  DoD  has  made  it  its 
goal  to  establish  performance  measures  in  the  IT  investment  process. 


19  Housel,  T.,  Jansen,  E.,  Pavlou,  P.  &  Rodgers,  W.  “Measuring  the  Return  on  Information 
Technology:  A  Knowledge-Based  Approach  for  Revenue  Allocation  at  the  Process  and  Firm  Level.” 
Monterey,  CA:  Naval  Postgraduate  School,  August  2005. 
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1.  Defining  IT  Performance  Measures  in  the  DoD 

The  Department  of  Defense  Guide  for  Managing  Information  Technology  (IT)  as 
an  Investment  and  Measuring  Perfonnance  v.1.0  (10  February  1997),  defines  IT 
performance  measure  as: 

the  assessment  of  effectiveness  and  efficiency  of  IT  in  support  of  the 
achievement  of  an  organization’s  missions,  goals,  and  quantitative 
objectives  through  the  application  of  outcome-based,  measurable,  and 
quantifiable  criteria,  compared  against  an  established  baseline,  to 
activities,  operations,  and  processes  [...]  Performance  Measurement  is  the 
means  by  which  an  organization  measures  it  effectiveness  and  efficiency 
in  the  pursuit  of  its  missions,  goals,  and  objectives.20 

Effectiveness  implies  accomplishing  the  required  mission  tasks  and  goals  while  ensuring 
customer  satisfaction  and  high  quality.  Efficiency  implies  that  the  best  possible  use  of 
available  resources  was  employed  to  improve  quantity  or  work,  reduce  costs  and  improve 
timeliness  of  the  overall  project. 

2.  Steps  for  Effective  Performance  Measurement 

To  assist  managers  in  maximizing  the  benefits  of  IT  investments,  the  guide 
outlines  six  steps  in  effective  performance  measurement:2 1 

•  Define  mission,  key  result  areas,  and  business  functions. 

•  Develop  mission  related  goals. 

•  Generate  performance  measures/indicators. 

•  Validate  and  verify  performance  measures. 

•  Implement  the  performance  measures  and  collect  data. 

•  Monitor  and  assess  the  results  and  repeat  the  process  as  needed. 

3.  Levels  of  Management 

The  DoD  also  recognizes  that  managers  at  different  levels  of  any  organization 
require  different  types  of  information  from  which  to  base  their  investment  and  business 

20  Department  of  Defense.  Guide  for  Managing  Information  Technology  (IT)  as  an  Investment  and 
Measuring  Performance.  Version  1.0.  Arlington,  Virginia:  Vector  Research  Inc.,  10  February  1997.  ES-i. 

21  Ibid. 
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strategies  on.  The  guide  identities  three  management  levels:  Enterprise,  Functional,  and 
Program/Project  Level.22  At  the  Enterprise  Level,  managers  are  focused  on  policy  and 
mission  decisions  to  guide  the  strategic  direction  of  acquisition  decisions.  The 
Functional  Level  managers  focus  on  unit  results  and  improving  the  operational  processes 
which  produce  them,  mission  related  outcome  measures  are  also  developed  at  this  level. 
At  the  Program/Project  Level,  managers  and  project  leaders  focus  on  the  activities  and 
tasks  critical  to  making  and  executing  tactical  and  management  decisions.  Program 
managers  and  project  leaders  are  also  the  focal  point  from  which  the  other  levels  receive 
detailed  performance  and  valuation  data  for  individual  systems,  projects,  processes  and 
activities.  This  is  the  level  where  the  CCOP  Program  Office  functions  and  where  this 
research  was  primarily  focused. 

4.  Performance  Measurement  at  the  Program/Project  Level 

Program/Project-Level  managers  and  leaders  capture  and  generate  the  information 
required  by  managers  at  the  other  two  levels.  It  is  necessary,  at  this  level,  to  determine 
what  key  information  is  needed  to  develop  defensible,  accurate  and  meaningful 
performance  measures.  Among  these  measures  is  Return  on  Investment,  ROI. 

At  a  minimum,  ROI  consists  of  both  a  return  (i.e.,  benefit)  component  and  an 
investment  (i.e.,  cost)  component.  The  cost  component  measures  the  number  of 
investment  dollars  management  has  placed  into  the  human,  infrastructure  and  IT  assets  of 
the  program  or  project;  this  infonnation  is  generally  documented  in  detail.  Benefit 
measures  are  much  more  complex.  The  DoD  guide  defines  “the  accomplishment  of 
functional  missions  and  goals,”22  as  a  measure  of  benefit.  How  a  program,  project, 
process  or  asset  contributes  to  overall  mission  accomplishment  is  normally  a  matter  of 
efficiency  and  effectiveness.  The  question  often  raised  though  is  “how  is  efficiency  and 
effectiveness  measured  and  how  do  they  translate  into  dollars?” 

To  be  mathematically  sound,  ROI  requires  that  benefits  AND  costs  be  monetized. 
While  mangers  or  analysts  can  directly  measure  the  benefits  of  some  investments  in 
monetary  tenns  (e.g.,  cost  savings  or  lower  overhead),  oftentimes  they  cannot  be  directly 

22  Department  of  Defense.  Guide  for  Managing  fnformation  Technology  (IT)  as  an  Investment  and 
Measuring  Performance.  Version  1.0.  Arlington,  Virginia:  Vector  Research  Inc.,  10  February  1997.  p.  3-4. 

22  Ibid.  6-9. 


13 


measured  monetarily.  Such  is  usually  the  case  with  many  IT  systems,  projects  and 
programs.  The  DoD  Guide  summarizes  12  approaches  to  measuring  the  performance  of 
IT  that  are  similar  to  those  in  Table  1,  but  recognizes  that  the  list  is  not  complete.24  The 
following  chapters  outline  an  alternative  method  and  approach. 


24  Department  of  Defense.  Guide  for  Managing  Information  Technology  (IT)  as  an  Investment  and 
Measuring  Performance.  Version  1.0.  Arlington,  Virginia:  Vector  Research  Inc.,  10  February  1997.  p.  8-1. 
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III.  THE  KNOWLEDGE  VALUE  ADDED  METHODOLOGY 


A.  THE  COMPARABILITY  AND  VALUE  PROBLEM 

There  are  several  situations  the  realm  of  business  and  non-profit  organizations 
that  can  benefit  from  a  method  for  measuring  the  value  of  knowledge  assets  both  human 
and  IT.  The  first  hurdle  in  solving  the  problem  is  defining  what  is  meant  by  “value”. 
The  simplest  measure  of  value,  as  presented  by  Dr.  Myron  L.  Cramer,  is  dollars:  how 
much  is  earned  or  saved.  The  metric  of  dollar-value  is  primarily  used  in  assessing 
contributions  in  a  commercial  or  business  context.25  A  problem  exists  though,  when 
applying  dollar-value  in  the  DoD  case  where  different  methods  are  used  to  assess  how 
systems  or  people  contribute  “value”  to  combat  effectiveness  or  mission  completion. 
These  methods  often  fail  to  capture  the  benefit  stream  produced  by  organizations, 
processes  or  assets  in  standard  or  comparable  units  of  measure.  And  oftentimes,  the  only 
metric  that  is  measured  with  any  level  of  standardization  and  accuracy  is  cost.  Benefits, 
while  measured  in  terms  of  varying  degrees  of  contribution,  are  seldom  represented  in  the 
monetized  fashion  required  by  traditional  accounting  methods  for  extracting  a  value  or 
Return  on  Investment. 

Another  problem  present  in  the  DoD  case  is  the  translation  of  outputs  into  monetary 
benefits.  Whereas  in  the  commercial  case,  a  price  per  unit  is  assigned  to  the  outputs, 
there  is  no  equivalent  pricing  mechanism  in  the  DoD  or  non-profit  case.  This  presents  a 
problem  when  conducting  empirical  financial  analysis  and  in  particular  when  seeking  a 
baseline  from  which  to  formulate  sound  fiscal  decisions. 

B.  THE  KVA  SOLUTION 

1.  Knowledge  Value-Added  (KVA)  Theory 

Knowledge  Value-Added  (KVA)  created  by  Dr.  Tom  Housel  (Naval  Postgraduate 
School)  and  Dr.  Valery  Kanevsky  (Agilent  Labs).  Initiated  almost  15  years  ago,  KVA 
was  built  on  tenets  of  complexity  and  entropy  theory,  and  was  developed  in  response  to 
business  process  reengineering  efforts  of  organizations  that,  with  no  other  method 

25  Cramer,  Myron.  “Measuring  the  Value  of  Information.”  Atlanta,  GA:  Georgia  Tech  Research 
Institute,  <http://iw.gtri.gatech.edu>  [20  July  2005].  p.  2 
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available,  focused  primarily  on  cost  cutting  measures  to  detennine  the  “value”  of  a 
process  or  knowledge  asset.  KVA  is  based  on  the  assumption  that  humans  and 
technology  in  organizations  add  value  by  taking  inputs  and  changing  them  into  outputs 
through  core  processes.2^  The  amount  of  change  an  asset  or  process  produces  can,  in 
fact,  be  a  measure  of  value  or  benefit. 

KVA  is  a  general  theory  for  estimating  the  value  added  by  knowledge  assets, 
human  and  IT,  using  a  methodology  that  is  analytic  and  tautological.  It  is  based  on  the 
premise  that  businesses  and  other  organizations  produce  outputs  (e.g.,  products  and 
services)  through  a  series  of  processes  and  subprocesses  which  change,  in  some  manner, 
the  raw  inputs  (i.e.,  labor  into  services,  information  into  reports).  KVA  explains  the 
changes  made  on  the  inputs  by  organizational  processes  to  produce  outputs  in  terms  of 
the  equivalent  corresponding  changes  in  entropy.  The  concept  of  entropy  is  defined  in 
the  American  Heritage  Dictionary  as  a  “measure  of  the  degree  of  disorder  [or  change]  in 
a  closed  system.”  In  the  business  context,  it  can  be  used  as  a  surrogate  for  the  amount  of 
changes  that  a  process  makes  to  inputs  to  produce  the  resulting  outputs.  22 

2.  KVA  Assumptions 

KVA  uses  knowledge-based  metaphor  to  operationalize  the  relationship  between 
change  in  entropy  and  value  added.  The  units  of  change  induced  by  a  process  to  produce 
an  output  are  described  in  terms  of  the  knowledge  required  to  make  the  changes.  More 
specifically,  the  time  it  takes  the  average  learner  “to  acquire  the  procedural  knowledge 
required  to  produce  a  process  output  provides  a  practical  surrogate  for  the  corresponding 
changes  in  entropy.”28  This  concept  is  illustrated  in  Figure  1. 


26  Housel,  T.  and  Bell,  A.  Measuring  and  Managing  Knowledge.  Boston:  McGraw-Hill,  2001.  pp.  92- 

93.  . . 

27  Housel,  T.  El  Sawy,  O.,  Zhong,  J.,  and  Rodgers,  W.  “Models  for  Mearsuring  the  Reutrn  on 
Information  Technology:  A  Proof  of  Concept  Demonstration.”  22nd  International  Conference  on  Inormation 
Systems.  December,  2001.  p.  13. 

28  Ibid. 
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P(X)  =  y 


Process  Pis  a  business  process. 

If  input  x  is  equal  to  output.?,  no  value  has  been  added  by  process  P 

Ifinputxis  changed  by  process  Pinto  output.?,  value  has  been  added  and  isoc  change. 

Change  can  be  measured  by  the  amount  of  knowledge  required  to  make  the  change. 

This  knowledge«the  amount  of  time  ittakesforan  average  learnerto  acquire  the  knowledge. 
So,  value  added  by  process ss  changes  knowledge  required  to  produce  the  outputs^  K 


Figure  1.  The  Housel-Kanevsky  Value-Added  Cycle. 29 


KVA  theory  also  assumes  that  all  process  outputs  can  be  described  in  equivalent 
units  (e.g.,  the  time  required  to  learn  how  to  produce  outputs).  The  advantages  of  this 
level  of  process  output  description  include:30 

•  The  ability  to  compare  all  processes  in  terms  of  their  relative  productivity 

•  The  ability  to  allocate  revenue  to  a  common  unit  of  output 

•  The  ability  to  describe  the  value  added  by  IT  in  terms  of  the  outputs  it 
produces 

•  The  ability  to  relate  outputs  to  the  cost  to  produce  those  outputs  in 
common  units 

•  A  common  unit  of  measure  for  organizational  productivity 

By  describing  all  process  outputs  in  common  units  it  is  possible  for  managers  and 
analysts  to  assign  revenue,  as  well  as  cost,  to  those  units  at  any  given  point  in  time.  With 
the  resulting  information,  traditional  accounting  and  financial  performance  and 
profitability  metrics  can  be  applied  at  the  sub-organizational  level. 

Other  assumptions  include  the  concept  of  Learning  Time.  Learning  Time  is 
measured  in  common  units  of  time  and  is  a  surrogate  for  the  procedural  knowledge 
required  to  produce  process  outputs.  Units  of  Learning  Time  represent  common  units  of 
output  which  are  described  by  the  variable  K.  Procedural  knowledge  can  be  embedded  in 
process  assets  such  as  IT,  employees,  training  manuals,  etc.  A  single  execution  of  a 
process  is  equivalent  to  a  single  unit  of  output,  represented  by  a  given  number  of 


29  Housel  and  Bell.  2001. 

30  Housel,  et  al.  December  2001.  p.  11. 
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common  units,  K.  Additional  levels  of  detail  in  process  descriptions  provide  additional 
levels  of  accuracy  in  the  estimation  of  the  number  of  knowledge  units  comprising  those 
processes. 

3.  Three  Approaches  to  KVA 

In  addition  to  the  Learning  Time  approach  for  calculating  KVA,  Table  2 
illustrates  two  other  methods  which  can  be  used. 


Learning  Time 

Approach 

Process  Description 
Approach 

Binary  Query  Method 

Identify  compound 
process  and  its 
component  processes. 

Identify  compound 
process  and  its 
component  processes. 

Identify  compound 
process  and  its 
component  processes. 

Establish  common  units  to 
measure  learning  time. 

Describe  the  products  in 
terms  of  the  instructions 
required  to  reproduce 
them  and  select  unit  of 
process  description 

Create  a  set  of  binary 
YES/NO  questions  such 
that  all  possible  outputs 
are  represented  as  a 
sequence  of  YES/NO 
answers. 

Calculate  learning  time  to 
execute  each  component 
process. 

Calculate  number  of 
process  description  words, 
pages  in  manual,  lines  of 
computer  code  pertaining 
to  each  process. 

Calculate  length  of 
sequence  of  YES/NO 
answers  for  each 
component  processes. 

Designate  sampling  time 
period  long  enough  to 
capture  a  representative 
sample  of  the  compound 
processes'  final 
product/service  output. 

Designate  sampling  time 
period  long  enough  to 
capture  a  representative 
sample  of  the  compound 
processes'  final 
product/service  output. 

Designate  sampling  time 
period  long  enough  to 
capture  a  representative 
sample  of  the  compound 
processes'  final 
product/service  output. 

Multiply  the  learning  time 
for  each  component 
process  by  the  number  of 
times  the  component 
executes  during  sample 
period. 

Multiply  the  number  of 
process  words  used  to 
describe  each  component 
process  by  the  number  of 
times  the  component 
executes  during  sample 
period. 

Multiply  the  length  of  the 
YES/NO  string  for  each 
component  process  by  the 
number  of  times  this 
component  executes 
during  sample  period. 

Allocate  revenue  to 
component  processes  in 
proportion  to  the 
quantities  generated  by 
previous  step. 

Allocate  revenue  to 
component  processes  in 
proportion  to  the 
quantities  generated  by 
previous  step. 

Allocate  revenue  to 
component  processes  in 
proportion  to  the 
quantities  generated  by 
previous  step. 
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Calculate  the  cost  to 
execute  each  component 
process,  calculate  return 
on  investment  per  process 
by  dividing  revenue 
allocated  to  component 
process  by  cost  of 
component  process. 


Calculate  the  cost  to 
execute  each  component 
process,  calculate  return 
on  investment  per  process 
by  dividing  revenue 
allocated  to  component 
process  by  cost  of 
component  process. 


Calculate  the  cost  to 
execute  each  component 
process,  calculate  return 
on  investment  per  process 
by  dividing  revenue 
allocated  to  component 
process  by  cost  of 
component  process. 


Table  2.  Three  Approaches  of  KVA. 3 1 


The  procedural  steps  of  KVA  as  applied  in  this  research  are  outlined  in  the  following 
case  study. 


31  Walsh,  David.  “Knowledge  Value  Added:  Assessing  both  Fixed  and  Variable  Value.”  Business 
Process  Audits. Com.  White  Papers.  Business  Process  Audits. Com,  13  August  1998. 
<http://www.businessprocessaudits.com/kvawalsh.com>  [06  June  2005] 
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IV.  METHODOLOGY  CASE  STUDY:  USS  READINESS 


A.  INTRODUCTION 

In  late  2004,  the  staff  of  the  United  States  Navy’s  Chief  of  Naval  Operations 
(OPNAV)  directed  its  Cryptologic  Carry-On  Program  Office  (OPNAV  N201)  to  focus  on 
three  goals  for  the  following  fiscal  year:  Efficiencies,  Metrics,  and  Return  on 
Investment.32  Measuring  efficiencies,  developing  proper  metrics,  and  determining  return 
on  investment  of  the  CCOP  portfolio  of  Information  Warfare  (IW)  systems  had,  for  many 
years,  seemed  a  bridge  too  far.  Faced  with  this  challenge  and  with  the  yearly  challenge 
of  deciding  how  to  allocate  precious  funding  dollars  to  the  twelve  IW  systems  which 
currently  make  up  the  CCOP  portfolio,  Lieutenant  Commander  Brian  Prevo,  the  CCOP 
program  officer  had  to  answer  some  fundamental  questions.  Should  he  merely  give  each 
an  equal  amount  of  continuous  funding?  Should  he  ask  which  ones  needed  the  most 
funding  to  continue  or  upgrade?  Should  he  ask  the  users  which  ones  they  liked?  None  of 
these  questions  really  went  to  the  heart  of  the  problem  from  his  perspective.  He  really 
wanted  a  relatively  objective  estimate  of  the  contribution  of  performance  of  each.  The 
simple  question  for  him  to  answer  before  he  made  his  next  moves  was,  “What  is  the 
return  on  investment  of  each  CCOP  system?”  Further,  he  wanted  to  know  the  best 
strategic  “options”  for  the  employment  and  deployment  of  his  systems.  He  wanted  to 
know  where  his  investments  might  provide  the  best  future  perfonnance  in  supporting  the 
Navy’s  Global  Intelligence,  Surveillance,  and  Reconnaissance  (ISR)  mission. 

LCDR  Prevo  turned  to  a  team  from  the  Naval  Postgraduate  School’s  (NPS) 
Graduate  School  of  Operations  and  Information  Sciences,  Dr.’s  Tom  Housel  and 
Johnathan  Mun,  to  help  him  devise  a  way  to  allocate  his  limited  funding  resources.  He 
also  enlisted  the  help  of  a  fellow  Naval  Cryptologist  and  Information  Warfare  Officer 
who  happened  to  be  a  student  at  NPS,  LCDR  Cesar  Rios.  Rios  had  operated  CCOP 
systems  and  other  IW  systems  while  conducting  ISR  missions  from  various  Navy 
platforms,  to  include  ships  and  aircraft.  As  team  lead  and  subject  matter  expert,  LCDR 
Rios  sought  advice  from  Dr.’s  Housel  and  Mun  about  how  to  structure  this  problem  in  a 

32  Department  of  the  Navy.  CCOP  Program  Briefing.  Power  Point.  Washington:  Dept,  of  the  Navy, 
CCOP  Program  Office  (OPNAV  N201C),  25  April  2005. 
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way  that  would  provide  the  ROI  and  options  information  that  the  CCOP  Program  Office 
needed  to  make  resource  allocation  decisions.  Mun  would  provide  the  real  options  (RO) 
analysis  approach  which  would  be  conducted  in  follow-on  to  this  research.  The 
Knowledge  Value  Added  (KVA)  approach  provides  the  raw  data  necessary  for  this  kind 
of  analysis.  Housel  is  an  expert  in  the  KVA  methodology.  KVA  provides  a  method  for 
describing  the  output  of  all  process  assets  (human  and  information  technology)  in 
common  units.  Market  comparables  can  then  be  generated  from  the  commercial  world  to 
estimate  the  price  per  common  unit,  which  allows  revenue  estimates  for  the  process 
outputs  of  non-profit  organizations  such  as  the  U.S.  Navy.  It  also  allows  a  cost  per  unit 
of  output  that,  when  combined  with  the  revenue  estimates,  provides  the  raw  data  for  ROI 
analysis. 

Armed  with  these  methodologies,  the  team  began  the  process  of  collecting  the 
data  required  to  provide  LCDR  Prevo  with  the  analysis  he  needed  to  support  is  CCOP 
resource  allocation  decisions.  The  following  is  a  synopsis  of  the  research,  including  the 
use  of  new  software  that  provides  KVA  and  RO  analysis. 

1.  Objective 

The  overall  objective  of  the  research  was  to  develop  a  model  and  methodology  to 
assist  in  the  CCOP  Program  Office’s  budgeting  process  for  the  acquisition  of  information 
warfare  systems.  The  model  was  to  assess  the  effectiveness  and  efficiency  of  the  systems 
in  support  of  the  achievement  of  the  Navy  Intelligence,  Surveillance,  and  Reconnaissance 
(ISR)  mission.  The  methodology  had  to  be  capable  of  producing  measurable  objectives 
that  were  reliable  and  based  on  clearly  defined  criteria.  Once  measures  were  defined,  an 
objective  perfonnance  baseline  for  each  system  in  the  portfolio  could  be  created  with 
which  to  compare  the  impact  of  existing  and  future  CCOP  systems  on  ISR  activities, 
processes  and  operations.  The  resulting  infonnation  could  then  be  utilized  to  make 
grounded  and  defensible  financial  decisions  and  projections  in  the  acquisition  and 
deployment  of  these  systems. 

2.  Method 

This  case  study  demonstrated  the  operationalization  of  the  Knowledge  Value 
Added  (KVA)  Methodology’s  ability  to  extract  the  measurable  and  quantifiable  data 
required  to  develop  a  performance  baseline  for  valuing  the  impact  of  CCOP  systems  on 
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the  process  in  which  they  are  employed,  the  Intelligence  Collection  Process  (ICP).  In 
particular,  this  case  focused  on  the  shipbome  ICP  and  the  performance  of  the  human  and 
information  technology  (IT)  components  that  comprise  the  process. 

B.  HYPOTHESIS 

The  Performance  metrics  of  CCOP  systems  operating  within  the  ICP  can  be 
derived  by  applying  the  Knowledge  Value  Added  Methodology.  These  metrics  can  be 
used  to  apply  financial  estimates  for  the  determination  of  Return  on  Investment,  Return 
on  Assets  and  Return  on  Performance. 

C.  ANALYSIS  AND  DATA  COLLECTION 

1.  The  ICP  and  CCOP 

To  accurately  measure  the  performance  of  CCOP  systems  it  is  necessary  to 
observe  the  systems  as  they  operate  within  the  overall  ICP  along  with  the  fixed 
infrastructure  and  human  operators.  Although  the  ICP  differs  slightly  between  the 
multiple  collection  platforms  (i.e.,  aircraft,  destroyers,  cruisers,  etc.),  the  process  will  be 
described  at  a  general-enough  level  to  achieve  an  aggregate  view  of  the  process  elements 
and  their  outputs. 

CCOP  systems  have  a  broad  scope  of  capabilities  and  functions.  Within  the  ICP, 
they  may  be  used  to  perform  operations  in  a  single  subprocess  or  in  multiple 
subprocesses  simultaneously.  CCOP  systems  may  be  used  in  a  stand-alone  manner,  or 
they  may  augment  existing  infrastructure  or  capabilities  onboard  the  ISR  platform.  The 
systems  are  also  capable  of  performing  fully  automated  functions  as  well  as  human- 
enabled  functions.  It  is  this  particular  mix  of  dynamic  behaviors  that  makes  analyzing 
CCOP  systems  a  challenge. 

2.  The  Data  Collection  Challenge 

The  ISR  Mission  is  generally  conducted  at  a  highly  classified  level.  Many 
particulars  of  the  ICP  and  the  CCOP  systems  and  their  outputs  are  closely  guarded 
information.  Deriving  the  information  required  to  perform  analysis  at  the  unclassified 
level  is  difficult,  and  at  times,  impossible.  For  the  purpose  of  academic  research,  much 
of  the  data  was  estimated  or  inferred  based  on  realistic  sampling  of  unclassified  process 
information.  Infonnation  on  human  capital,  such  as  salaries  and  operator  training,  are 
public  information  and  were  gathered  from  sources  such  as  the  Stay  Navy  Website  and 
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the  Center  for  Information  Dominance  (CID)  training  documentation.  The  equipment 
data  was  also  derived  or  inferred  from  documentation  provided  by  the  OPNAV  N20  staff 
and  the  Space  and  Naval  Warfare  Command  (SPAWAR).  Other  information  such  as 
number  of  process  outputs  and  executions  were  extrapolated  from  samples  gathered  via 
interviews  with  ISR  crews  currently  operating  onboard,  deployed  U.S.  Navy  surface 
ships. 

D.  MODELING  THE  USS  READINESS  ICP 

The  intelligence  collection  process  (ICP)  is  the  means  by  which  tactical  Navy 
intelligence,  surveillance  and  reconnaissance  (ISR)  units  (i.e.,  ships,  aircraft  and  other 
platfonns)  receive  requests  for  intelligence  or  intelligence  collection  tasking  and  apply 
various  human  disciplines  and  IT  technologies  to  search,  acquire,  process  and  report 
results  back  to  tactical  users  (i.e.,  fleet  staffs  and  strike  groups),  and  national-level 
consumers  (i.e.,  NSA).  The  following  table  is  a  brief  description  of  the  generalized 


process: 


1- 

Sub-Process  Name 

Sub-Process 

Description 

PI 

Review  Request 

•  Determine  if  collection  capability  is  available 

•  Determine  if  further  direction  or  info  required 

P2 

Determine  Op/Equip  Mix 

•  Review  directives  and  target  information  to 
determine  type/category  of  target 

P3 

Input  Search/Function  into  CCOP 

•  Assign  search  blocks  and  allocate  system 
resources  to  each  target 

P4 

Search/Collection  Process 

•  Targeted  or  full  spectrum  search 

•  Observe  sensor  data  for  target  cues 

P5 

Target  Data  Acquisition/Capture 

•  Audio  Routing 

•  Record/Capture  Data 

P6 

Target  Data  Processing 

•  Demodulate,  decrypt,  direction  find  (DF),  or 
Geo-locate 

•  Translate 

P7 

Target  Data  Analysis 

•  Human  or  IT-based  analysis  of  captured  data 

P8 

Format  Data  for  Report 

Generation 

•  Input  data  into  required  reporting  formats 

P9 

QC  Report 

•  Check  format,  accuracy  and  adherence  to 
tasking,  regulations  and  laws 

P10 

Transmit  Report 

•  Transmit  via  secure  voice  radio,  secure 
internet  relay  chat,  US  Message  Traffic 

Format 

Table  3.  The  Intelligence  Collection  Process  (ICP). 
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Each  sub-process  can  be  further  broken  down  into  individual  actions  that  may  be 
required  to  perform  the  subprocess.  Below  is  the  breakdown  of  sub-process  P6  Target 
Data  Processing : 


P6 

Target  Data  Processing 

Human-based  (no  automation  required) 

Manual  copy  directly  into  report 

Human  translation  &  processing 

IT-based 

Direct  transfer  into  report 

Demodulate 

All  IT-based 

Human-enabled 

Decrypt 

All  IT-based 

Human-enabled 

Direction  finding 

Automatic  -  Local  Line  Of  Bearing  (LOB) 

Human-enabled  -  local  LOB 

Human-enabled  -  B-rep  request 

Geolocation 

Special  processing 

Table  4.  Process  P6  Activities. 


1.  USS  READINESS 

USS  READINESS  is  a  fictitious  U.S.  Navy  warship  which  has  been  outfitted  for 
conducting  ISR  missions.  Along  with  the  general  manning,  the  ship  has  a  contingent  of 
Information  Warfare  operators  who  perfonn  the  ICP  utilizing  CCOP  systems  that 
interconnect  with  a  pre-existing  IT  infrastructure.  For  the  purpose  of  this  research,  it  is 
assumed  that  the  ship  is  on  a  typical  six-month  deployment  and  receives  daily  tasking  for 
ISR  collection  from  higher  authorities  as  well  as  intelligence  requirements  from  the 
commanding  officer  and  group  commanders.  In  response  to  tasking,  the  ISR  crew 
onboard  USS  READINESS  produces  a  variety  of  reports  which  include  raw-intelligence 
reports,  technical  reports,  analyst-to-analyst  exchanges  and  daily  collection  summaries. 
The  table  below  illustrates  the  ISR  crew  and  the  subprocesses  within  the  ICP  in  which 
they  perform: 
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IW  Operator 

Assigned  to  ICP 
Processes 

Division  Officer 

1,2,9 

Division  Leading  Petty 

Officer 

1,2, 9, 7 

Signals  Operator  1 

3-7,9 

Signals  Operator  2 

3-7 

Signals  Operator  3 

4-7 

Signals  Operator  4 

4-7 

Signals  Operator  5 

4-7 

Comms  Operator  1 

8,10 

Comms  Operator  2 

8,10 

Comms  Operator  3 

8,10 

Table  5.  USS  READIN 

ESS  ISR  Crew. 

USS  READINESS  is  outfitted  with  four  CCOP  systems  (A,  B,  C,  and  D)  which 
operate  in  the  following  subprocesses: 


Subprocess  Name 

CCOP 

Assigned 

pi 

Review  Request/Tasking 

A 

P2 

Determine  Op/Equip  Mix 

A 

P3 

Input  Search  Function/Coverage  Plan 

A 

P4 

Search/Collection  Process 

A,B 

P5 

Target  Data  Acquisition/Capture 

A,B 

P6 

Target  Data  Processing 

A,B,C,D 

P7 

Target  Data  Analysis 

A,C,D 

P8 

Format  Data  for  Report  Generation 

A 

P9 

QC  Report 

A 

P10 

Transmit  Report 

A 

Table  6.  USS  READINESS  CCOP  Systems. 


As  shown  in  Table  6,  CCOP  systems  may  be  used  in  a  single  subprocess  or  across 
multiple  subprocesses  along  with  the  existing  infrastructure  available  in  each  particular 
platform.  Additionally,  some  systems  such  as  CCOP  A  are  highly  complex  and 
comprised  of  multiple  subsystems.  Below  is  a  breakdown  of  CCOP  A  and  its  related 
components. 
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CCOP  A  (Example) 

Component 

Description/Functions 

Radio  Frequency  Management  System 

•  RF  management 

Signal  Acquisition  System 

•  Energy  Search 

Audio  Distribution  System 

•  Audio  Routing  &  Recording 

Intermediate  Frequency  Signal  Processing 

System 

•  Spectrum  Display  Operations 

•  Signal  Processing  Applications 

Control  &  Processing  System 

•  Coverage  Plan  Creation/Management 

Common  Cryptologic  Workstation  (CCWS) 

•  Database  Operations 

•  JMCIS  Applications 

•  Cryptologic  Unified  Build  Applications 

•  Microsoft  Applications 

•  Signal  Processing  Applications 

Table  7.  CCOP  A  Components. 


To  accurately  capture  the  contribution  of  CCOP  systems  across  subprocesses  it  is 
necessary  to  break  down  each  system  to  its  individual  components  and  functions,  analyze 
the  attributes  of  each,  and  insert  them  into  the  appropriate  subprocess  of  the  ICP. 

E.  APPLYING  KVA 

As  described  in  the  previous  chapter,  KVA  uses  a  knowledge-based  metaphor  as  a 
means  to  describe  units  of  change  in  tenns  of  the  knowledge  required  to  make  the 
changes.  The  analysis  is  accomplished  by  extracting  historical  data  to  build  metrics  that 
are  structured  in  the  same  manner  as  common  financial  metrics.  The  difference  between 
KVA  and  other  financial  models  though,  is  that  KVA  provides  the  means  to  analyze  the 
metrics  at  a  sub-organizational  level  and  allows  for  the  allocation  of  cost  and  revenue 
across  subprocesses  for  accounting  purposes. 

1.  KVA  Assumptions 

The  underlying  assumptions  of  KVA  are  that: 

•  Human  and  technology  in  organizations  take  inputs  and  change  them  into 
outputs  through  core  processes 

•  By  describing  all  process  outputs  in  common  units  (i.e.,  the  knowledge 
required  to  produce  the  outputs)  it  is  possible  to  assign  revenue,  as  well  as 
cost,  to  those  processes  at  any  given  point  in  time. 

•  All  outputs  can  be  described  in  terms  of  the  time  required  to  leam  how  to 
produce  them. 

•  Learning  Time  is  measured  in  common  units  of  time  and  is  also  a 
surrogate  for  knowledge.  Thus,  units  of  Learning  Time  can  also  be  called 
Common  Units  of  Output  (K). 
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•  Having  a  common  unit  of  output  makes  it  possible  to  compare  all  outputs 
in  terms  of  cost  per  unit  as  well  as  price  per  unit,  since  revenue  can  now 
be  assigned  at  the  sub-organizational  level. 

•  Once  cost  and  revenue  stream  have  been  assigned  to  sub-organizational 
outputs,  nonnal  accounting  and  financial  performance  and  profitability 
metrics  can  be  applied  to  them. 

2.  Case  Study  Assumptions  and  Data 

The  following  assumptions  and  data  apply  to  the  USS  READINESS  -  KVA  proof 
of  concept  case  study.  Figures  and  results  are  estimates  only,  and  should  not  be  viewed 
as  projections  of  actual  capabilities. 

a.  Assumptions 

Length  of  Sample  Period :  The  sample  period  for  this  analysis  was  one 
deployment  period  (6  months).  For  this  reason,  some  annual  cost  data  is  adjusted  to 
reflect  the  sample  period. 

Cost  Assumptions'.  Cost  of  human  capital  assets  was  derived  from  annual 
U.S.  Navy  salary  information  available  on  the  Stay  Navy  website  for  Fiscal  Year  2005 
with  allowances  calculated  for  a  crew  home  ported  in  San  Diego,  CA.  Operator  rank  and 
bonus  level  information  was  estimated  based  on  a  typical  shipbome  ISR  crew. 
Equipment  costs  were  derived  from  annual  cost  data  provided  by  the  OPNAV  N20  staff. 
While  equipment  costs  reflect  installation  and  training  costs,  they  do  not  reflect 
amortization  or  the  total  operational  cost  of  the  program. 

Proxy  Revenue  Assumptions'.  Proxy  revenues  are  based  on  the  following 
assumption:  if  a  commercial  entity  or  organization  produces  comparable  outputs  as  a 
not-for-profit  organization  such  as  the  Department  of  Defense,  and  the  processes  required 
to  produce  those  outputs  are  comparable,  certain  inferences  can  be  derived.  First,  if  the 
processes  are  comparable,  the  outputs  of  the  commercial  case  are  comparable  to  the 
outputs  of  the  not-for-profit  case.  Second,  if  market  forces  have  placed  a  “value”  or 
price-per-unit  to  the  comparable  commercial  outputs  yielding  a  revenue  stream  for  the 
commercial  entity,  that  price-per-unit  can  also  be  applied  to  the  not-for-profit  case. 
Lastly,  the  derived  price-per-unit  can  be  used  to  develop  an  analytical  or  hypothetical 
revenue  stream  for  the  not-for-profit  organization.  For  this  research,  the  price-per-report 
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of  nine  competitive/business  intelligence  organizations  and  private  military  corporations 
were  averaged  to  yield  a  notional  market  comparable  price-per-unit  of  the  outputs  of  the 
USS  READINESS  ICP.  Figure  2  illustrates  two  of  the  corporations  sampled  in  the 
Intelligence  Price-per-Unit  study. 


IHL  Consulting  Group 

IHL  Consulting  Group  is  a  global  research  and  advisory  firm  specializing  in 
technologies  for  the  retail  and  hospitality  industries.  The  company,  generates 
timely  data  reports,  offers  advisory  services  and  serves  as  the  leading  retail 
technology  spokesperson  for  industry  and  vendor  events.  IHL  provides  customized 
business  intelligence  for  retailers  and  retail  technology  vendors,  with  particular 
expertise  in  supply  chain  and  store  level  systems.  Their  customers  are  retailers 
and  retail  technology  providers  who  want  to  better  understand  what  is  going  on  in 
the  overall  technology  market,  or  wish  to  identify  specific  equipment  needs  for  the 
retail  market. 

IHL’s  price  per  report  ranges  from  $1,495  to  $3,295 

J.C.  Owens  Global  Consulting,  LLC 

J.C.  Owens  Global  Consulting,  LLC  is  a  subsidiary  of  J.  C.  Owens  Group 
Worldwide.  It  has  been  established  to  provide  international  investigation,  business 
intelligence  and  risk  consulting  services  to  corporate  organizations  and  government 
agencies  worldwide  in  1 82  countries  around  the  world.  Their  specific  areas  of  capability 
include:  Global  Corporate  Investigations;  Global  Business  Intelligence;  Intellectual 
Property/Copyright  Trademark  Investigations;  Background/Pre-employment/Due 
Diligence/Litigation  Support;  Insurance  Fraud  &  Claim  Investigation. 

The  firm  presently  covers  a  total  of  182  countries  in  Africa,  North/South 
America,  Middle  East,  Central  America,  Europe,  Asia,  and  the  Caribbean.  It  maintains 
an  office  in  Bloomfield,  New  Jersey,  United  States  of  America,  from  where  it  covers  the 
world;  and  an  office  in  Lagos,  Nigeria,  which  handles  its  African  operations. 

Price  per  Global  Intelligence  Report/Assignment  is  approximately  $5,000  (US) 


Figure  2.  Intelligence  Price-per-Unit  Benchmarking  Sample. 33 

Output  Assumptions:  Although  the  ICP  onboard  the  ISR  unit  produces  a 
variety  of  reports,  only  raw-intelligence  reports  (also  known  as  Klieghlights  or  KL)  were 
counted  as  outputs  for  this  research.  It  was  also  assumed  that  each  report  was  of  equal 
tactical  value/weight  and  importance,  in  essence,  all  outputs  were  similar. 

Other  Assumptions :  IT  Learning  Time.  It  was  assumed  that  the 
knowledge  embedded  in  information  technology  (IT)  systems  can  be  derived  by 
averaging  the  time  it  would  take  an  average  learner  to  learn  how  to  produce  the  outputs 

produced  by  the  IT  systems  in  a  single  subprocess  output  cycle.  CCOP  systems  are 

33  Data  for  IHL  Consulting  Group  was  gathered  from  the  2005  IGL  Consulting  Group  Research  Price 
List  which  is  available  from  www.ihlservices.com 

Data  for  J.C.  Owens  Global  Consulting,  LLC  was  furnished  by  Mr.  Israel  Mbachu,  CFE,  CII,  Principal 
Partner  at  J.C.  Owens  Global  Consulting,  LLC.  Email  dated  09  September  2005. 


29 


highly  complex  and  at  times,  comprise  multiple  components  with  varying  functions.  To 
estimate  the  time  to  leam  of  a  single  CCOP  system,  the  components  were  analyzed 
individually.  Academic  authorities  on  the  functions  performed  by  each  were  consulted  to 
determine  the  length  of  time  it  would  take  an  average  learner  (assuming  at  least  a 
Bachelor’s  of  Science  (B.S.)  degree  in  Electrical  Engineering  or  in  a  field  related  to  the 
component)  to  learn  how  to  produce  the  IT  outputs.  In  this  case,  subject  matter  experts  in 
the  functional  fields  of  each  system  were  consulted  to  estimate  the  IT  time  to  learn. 
Figure  3  illustrates  one  such  interview. 


CCOP  C  Learning  Time  Derivation  Example 

To  determine  the  learning  time  of  CCOP  C,  the  team  first  dissected  the  system  into  its  basic 
functional  components.  CCOP  C  is  the  AN/SSQ-120(V)  Transportable-Radio  Direction  Finder 
(T-RDF).  T-RDF  provides  a  low-cost  Medium/High/Very  High/Ultra  High  Frequency 
(MF/HF/VHF/UHF)  Direction  Finding  (DF)  capability  to  selected  U.S.  Navy  ships.  T-RDF  has 
two  major  components,  the  receiving  equipment  and  the  processing  unit. 

To  analyze  the  system  and  determine  its  time  to  learn,  the  team  consulted  Dr.  Richard  Adler, 
an  authority  on  signals  intelligence  (SIGINT)  systems  and  antenna  technologies.  It  was 
assumed  that,  as  a  baseline,  the  “average  learner”  to  be  taught  the  functions  of  T-RDF  would 
have  an  undergraduate  degree  in  a  related  technical  field  such  as  Electrical  Engineering. 

Dr.  Adler  suggested  that  the  underlying  disciplines  that  would  have  to  be  learned  are: 

-Basic  RF  Theory  (66  days) 

-EM  Theory/Formal  EM  (198  days) 

-Basic  Communications  Theory  (132  days) 

-Propagation  Theory  (66  days) 

-Antenna  Theory  (66  days) 

-Basic  Radio  Direction  Finding  (66  days) 

Aggregating  the  results,  an  estimate  of  594  days  of  learning  time  would  be  required  for  the 
average  learner  to  learn  how  to  produce  the  outputs  of  CCOP  C. 


Figure  3.  CCOP  C  Learning  Time.34  35 


34  Department  of  the  Navy.  Vision. .  .Presence. .  .Power:  A  Program  Guide  to  the  U.S.  Navy  -  2002 
Edition.  Washington:  Dept,  of  the  Navy,  2002.  Chapter  3. 

35  Dr.  Richard  Adler  is  a  Research  Associate  Professor  in  Department  of  Electrical  and  Computer 
Engineering  at  the  Naval  Postgraduate  School.  He  also  holds  positions  in  the  Research  Committee  and  is 
the  Supervisor  of  the  Signal  Enhancement  Lab.  Dr.  Adler  has  3 1  years  of  experience  in  undergraduate  and 
graduate  teaching  and  thesis  advising,  29  years  in  design  and  analysis  of  VLF-UHF  tactical,  strategic,  DF 
and  broadcast  antennas,  3 1  years  in  EM  numerical  analysis  of  the  effects  of  platforms  and  environment  on 
the  performance  of  antennas,  and  26  years  Hands-On-Workshops  on  Numerical  Antenna  Modeling  for  wire 
antennas,  reflector  antennas  and  general  scattering  shapes.  He  is  a  Registered  Professional  Engineer  in 
California. 
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b.  Human  Capital  Data 

Individual  data  sheets  were  created  for  each  member  of  the  fictitious  USS 
READINESS  ISR  crew.  The  information  included  the  typical  rank  and  subspecialty 
designators  or  Navy  Education  Codes  (NEC)  of  afloat-IW  operators.  Military  pay  was 
collected  along  with  NEC-specific  bonus  pay  for  each  operator.  Figure  4  illustrates  the 
amount  of  job  specific  schooling  and  on-the-job  training  received  by  each. 
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Figure  4.  Sample  Operator  Data  Sheets. 

The  following  tables  contain  the  actual  case  data  for  the  cost  of  human  assets  used  in  this 


research: 


Element 

Annual  Unit  Cost 
(avg) 

Annual  Unit 
Bonuses  (avg) 

Budget  (Cost)  per 
Sample  Pd 

Div  Officer 

$ 

79,298 

$ 

- 

$ 

39,649 

Div  LPO 

$ 

79,298 

$ 

- 

$ 

39,649 

SigOp  1 

$ 

61,883 

$ 

35,475 

$ 

48,679 

SigOp  2 

$ 

58,190 

$ 

14,958 

$ 

36,574 

SigOp  3 

$ 

44,391 

$ 

17,177 

$ 

30,784 

SigOp  4 

$ 

43,397 

$ 

- 

$ 

21,699 

SigOp  5 

$ 

41,518 

$ 

- 

$ 

20,759 

ComOpI 

$ 

44,391 

$ 

22,196 

ComOp2 

$ 

43,397 

$ 

21,699 

ComOp3 

$ 

41,518 

$ 

20,759 

Total  Human 

$ 

302,446 

Table  8 

USS  READIN] 

3SS  Human  Capital  Cost  Data. 
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The  total  amount  of  days  of  on-the-job  training  and  job  experience  of  the  human  assets 
are  shown  in  Table  9. 


Operator 

Time  in  Service 
(Days) 

Pre-Deployment 
Training  (Days) 

On-Job  Training 
(Days) 

Totals 

Div  Officer 

1,825 

15 

292 

2,132 

Div  LPO 

4,380 

15 

524 

4,919 

SigOp  1 

3,285 

30 

486 

3,801 

SigOp  2 

2,190 

30 

366 

2,586 

SigOp  3 

1,095 

30 

325 

1,450 

SigOp  4 

730 

30 

219 

979 

SigOp  5 

730 

30 

184 

944 

ComOpI 

2190 

20 

325 

2,535 

ComOp2 

1096 

20 

219 

1,335 

ComOp3 

730 

20 

184 

934 

Table  9.  USS  READINESS  Operator  Training  Time  (Days). 


c.  In  formation  Technology  Data 

Detailed  information  was  also  gathered  for  the  USS  READINESS  CCOP 
systems  and  fixed  infrastructure  analyzed  in  this  research.  Cost  data,  shown  below,  was 
derived  from  annual  budget  estimates. 


Element 

Avg  Annual  Unit 
Costs 

Budget 

(Cost)  per  Sample 
Pd 

CCOP  A 

$  193,500 

$ 

96,750 

CCOP  B 

$  44,643 

$ 

22,322 

CCOPC 

$  24,000 

$ 

12,000 

CCOP  D 

$  126,923 

$ 

63,462 

Total  CCOP 

$ 

194,533 

Fixed  IT  Infrastructure 

$ 

102,500 

TOTAL  IT 

$ 

297,033 

Table  10.  USS  READINESS  Systems  Cost  Data. 


To  obtain  the  time-to-learn  of  CCOP  systems,  each  was  analyzed  at  the  component  or 
subsystem-level.  The  learning-time  for  each  component  was  then  estimated  by 
interviewing  subject  matter  experts  and  deriving  how  many  days  it  would  take  the 
average  IW  operator  to  learn  how  to  perform  the  component  operation.  The  table  below 
contains  the  aggregated  estimations  used  for  each  system. 
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System 

Time  to  Learn 
(days) 

CCOP  A  Aggregated  Time  to  Learn  = 

4,092 

CCOP  B 

1,460 

CCOPC 

514 

CCOP  D 

1,825 

Table  11.  CCOP  System  Learning-' 

nine. 

3.  KVA  Steps 

The  following  is  a  summarized  synopsis  of  the  steps  in  using  the  KVA  method 
once  the  required  data  has  been  gathered.  Two  of  the  USS  READINESS  ICP 
subprocesses  are  used  to  illustrate  the  computations,  subprocesses  P4  and  P8.  Appendix 
A  contains  the  actual  analysis.  Each  step  also  contains  the  definitions  of  variables  and 
relevant  fonnulas  used. 

a.  Step  One:  Estimate  Process  Time-  to-Learn 

(1)  Definitions: 

Time  to  Learn  (tr)  is  the  time  it  would  take  the  average  learner  to  learn  how  to  produce 
a  single  subprocess  output. 

Human  Time  to  Learn  (Tlh)  is  the  time  it  would  take  the  average  learner  to  leam  the 
human-specific  portions  of  the  subprocess  required  to  produce  a  single  subprocess  output. 
In  this  case  factors  such  as  time-in-service,  schooling,  on  the  job  training,  and  pre¬ 
deployment  training  of  each  operator  were  used  to  estimate  the  human  time  to  learn. 

IT  Time  to  Learn  (tin)  is  the  time  it  would  take  the  same  average  learner  to  learn  how 
to  produce  the  outputs  produced  by  the  IT  systems  in  a  single  subprocess  output  cycle.  In 
this  case,  subject  matter  experts  in  the  functional  fields  of  each  system  were  consulted  to 
estimate  the  IT  time  to  learn  as  exampled  in  Figure  3. 

%  Automation  is  the  percent  of  a  process  that  is  automated  by  information  technology 
(IT)  over  and  above  human  contribution. 

(2)  Description:  First,  the  time  it  takes  an  average  Information 
Warfare  operator  to  learn  one  whole  subprocess  is  estimated.  This  represents  the  amount 
of  knowledge  required  to  produce  a  single  aggregate  subprocess  output.  Time  to  Leam 
(to  is  estimated  by  summing  the  average  time  spent  in  training  the  process  (pre¬ 
deployment  training),  the  combined  training  experience  of  each  operator  performing  the 
process  (Table  9.),  and  the  learning  time  embedded  in  each  IT  system  multiplied  by  the 
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percent  automation  of  the  particular  subprocess.  The  time  to  learn  of  CCOP  systems 
operating  in  more  than  one  subprocess  is  divided  equally  across  the  subprocess  in  which 
they  operate. 


Sub¬ 

process 

Name 

CCOP 

Assigned 

Process 

Training 

t[_H 

(days) 

Other 

Relevant 

t[_H 

(days) 

Tot  tLH 
(days) 

CCOP 

tur 

(days) 

Avg  % 
Automat'n 

Tot  tLrr 
times  % 
Automat'n 
(days) 

Tot  tLfor  1 
Process 
Output 
(days) 

P4 

Search  / 

Collection 

Process 

A,B 

35 

9760 

9795 

896 

46.25% 

414.34 

10,209.34 

P8 

Format 

Data  for 

Report 

Generatio 

n 

A 

10 

4804 

4814 

409 

71.67% 

293.27 

5,107.27 

Table  12.  P4  and  P8  Time  to  Learn. 


b.  Steps  Two  and  Three:  Calculate  the  K  Produced  by  IT  and 
Human  Assets.  Find  the  Total  K  for  Each  Subprocess 

(1)  Definitions: 

K  is  the  descriptive  tenn  chosen  for  the  common  units  of  output  estimated  by  KVA. 

Executions  (Ex)  are  the  average  number  of  times  a  process  asset,  human  or  IT,  produced 
an  individual  subprocess  output. 

Kh  is  the  common  units  of  output  attributed  to  human-asset  contribution. 

Kit  is  the  common  units  of  output  attributed  to  IT-asset  contribution. 

KP  is  the  total  common  units  of  output  for  each  subprocess. 

(2)  Formulas: 

Total  subprocess-asset  output:  KAsSet  =  (ExAsset)  (fi  ) 

Total  subprocess  output:  KP  =  KH  +  KiT 

Total  process  output:  KTOt  =  E(KP  ) 

(3)  Description:  The  number  of  times  an  element,  human  or  IT, 
executes  a  subprocess  is  tallied  and  then  multiplied  by  the  time  to  leam  of  the  particular 
subprocess.  The  resulting  number  is  the  amount  of  output,  by  asset,  used  in  the  execution 
of  the  subprocess.  Once  the  K  for  each  asset  has  been  calculated,  it  is  summed  to  reveal 
the  total  K  for  each  subprocess. 
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Total  Output  per  Subprocess  for  Sample  Period  - 
Including  Automation 

#of 

each 

asset 

per 

Posi¬ 

tion 

Asset 

# 

executns 
by  Asset 
P4 

Total  K 

P4 

# 

executns 
by  Asset 
P8 

Total  K 
P8 

1 

Div  Officer 

0.00 

0.00 

0.00 

0.00 

1 

Div  LPO 

0.00 

0.00 

0.00 

0.00 

1 

SigOp  1 

135.00 

1378260.68 

0.00 

0.00 

1 

SigOp  2 

144.00 

1470144.72 

0.00 

0.00 

1 

SigOp  3 

162.00 

1653912.81 

0.00 

0.00 

1 

SigOp  4 

152.00 

1551819.43 

0.00 

0.00 

1 

SigOp  5 

158.00 

1613075.46 

0.00 

0.00 

1 

ComOpI 

0.00 

0.00 

78.00 

398367.34 

1 

ComOp2 

0.00 

0.00 

78.00 

398367.34 

1 

ComOp3 

0.00 

0.00 

79.00 

403474.62 

1 

CCOP  A 

751.00 

7667213.09 

235.00 

1200209.31 

1 

CCOP  B 

92.00 

939259.13 

0.00 

0.00 

1 

CCOP  C 

0.00 

0.00 

0.00 

0.00 

1 

CCOP  D 

0.00 

0.00 

0.00 

0.00 

IT  Fixed 

1 

Infrastructure 

235.00 

2399194.51 

235.00 

1200209.31 

Kp 

18672879.81 

3600627.92 

Table  13.  P4  and  P8  Total  K  by  Asset. 


Subprocess  Name 

Kit 

(automation 
&  infras) 

Kh 

Kjot 

P4 

Search/Collection 

10,909,092.30 

7,599,933.50 

18,509,025.80 

P8 

Format  Data  for 
Report  Generation 

2,342,721.41 

1,171,360.71 

3,514,082.12 

Table  14.  P4  and  P8  IT/Human  Total  K. 


c.  Steps  Four  and  Five:  Derive  Proxy  Revenue  Stream  and 
Develop  the  Value  Equation  Numerator  by  Assigning  Revenue 
Streams  to  Subprocesses. 

(1)  Definitions: 

Market  Comparable  Price  per  Unit  is  the  notional  price  per  unit  allocated  to  the  outputs  of 
non-profit  organizations  based  on  the  market  price  per  unit  of  the  comparable  outputs  of  a 
similar  commercial  organization. 
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%  K  is  percent  of  the  total  K  produced  by  an  individual  subprocess  or  asset. 

(2)  Fonnulas: 

Proxy  Revenue:  Rtot  =  (Total  #  of  Process  Outputs)  (Market  Comp.  Price  per  Unit) 

%  of  Total  K  per  Subprocess:  %  KP  =  (KP  /  KTot)  x  100% 

Subprocess  Revenue  Allocation:  RP  =  %KP  x  RTot 


(3)  Description:  First,  utilizing  the  Market  Comparables  approach, 
the  total  number  of  ICP  outputs  is  multiplied  by  the  average  market  price-per-unit  to 


yield  a  Proxy  Revenue  for  the  USS  READINESS  ICP. 


Proxy  Revenue  Assumptions 


Market  Comparable  Price  Per  Unit  (avg) 
Avg#  Reports  executed/sample  pd 
Avg  Proxy  for  Revs  -  Sample  Pd  (Rtot) 


$  3,800 
235 

$  893,000 


Table  15.  USS  READINESS  ICP  Proxy  Revenue  Assumption. 


Next,  the  percent  of  the  total  process  K  produced  by  each 
subprocess  is  calculated. 


Subprocess  Name 

K for  IT 

(automation 
&  infras) 

Kfor 

Humans 

Total  K 

%  of  Total 

K  per  sub¬ 
process 

P4 

Search/Collection  io, 909, 092.30 

7,599,933.50 

18,509,025.80 

24.5398% 

P8 

Format  Data  for 

Report 

Generation  2,342,721.41 

1,171,360.71 

3,514,082.12 

4.6591% 

52,659,382.55 

22,765,206.13 

75,424,588.68 

100.0000% 

Table  16.  P4  and  P8  Percent  K. 


Revenues  can  now  be  assigned  to  subprocesses,  people  and  IT 
based  on  their  individual  %K: 
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Subprocess 

Name 

Kfor 

Humans 

Total  K 

%  of  Total 

K  per  sub¬ 
process 

Proxy 
Revenue 
Assigned 
to  Sub¬ 
process 
($US) 

%of 
Total  K 
for 

Human 
per  Sub¬ 
process 

Proxy 
Revenue 
Assigned 
to  Human 

K  ($US) 

P4 

Search/ 

Collection  7,599,933.50 

18,509,025.80 

24.5398% 

$219,140 

10.0762% 

$  89,980 

P8 

Format  Data  for 

Report 

Generation  1,171, 360.71 

3,514,082.12 

4.6591% 

$41,605 

1.5530% 

$  13,868 

22,765,206.13 

75,424,588.68 

100.0000% 

$  893,000 

30.1827% 

$  269,532 

Table  17.  P4  and  P8  Proxy  Revenue  Allocation  for  Human  Contribution. 


Subprocess  Name 

Kfor  IT 

(automation  & 
infras) 

Total  K 

%of 
Total  K 
for 

CCOP 

A 

Proxy 
Revenue 
Assigned  to 
CCOP  A 
Process  K 
($US) 

%of 
Total  K 
for 

CCOP 

B 

Proxy 
Revenue 
Assigned  to 
CCOP  B 
Process  K 
($US) 

P4 

Search/ 

Collection 

10,909,092.30 

18,509,025.80 

10.08% 

$  89,980 

1 .23% 

$  11,023 

P8 

Format  Data  for 
Report 

Generation 

2,342,721.41 

3,514,082.12 

1.55% 

$  13,868 

0.00% 

$  - 

52,659,382.55 

75,424,588.68 

30.18% 

$  269,532 

3.82% 

$34,110 

Table  18.  P4  and  P8  Proxy  Revenue  Allocation  for  CCOP  A  &  B  Contribution. 


Subprocess  Name 

%of 
Total  K 
for 

CCOP 

C 

Proxy 
Revenue 
Assigned  to 
CCOPC 
Process  K 
($US) 

%of 

Total 

Kfor 

CCOP 

D 

Proxy  Revenue 
Assigned  to 
CCOP  D 
Process  K 
($US) 

%  of  Total  K 
for  Fixed  IT 
Infrastructure 

Proxy 
Revenue 
Assigned 
to  Fixed 
Infras 
Process  K 
($US) 

P4 

Search  / 

Collection 

0.00% 

$ 

0.00% 

$ 

3.15% 

$  28,156 

P8 

Format  Data  for 
Report 

Generation 

0.00% 

$ 

0.00% 

$ 

1 .55% 

$  13,868 

6.52% 

$  58,253 

2.23% 

$  19,906 

27.06% 

$  241,667 

Table  19.  P4  and  P8  Proxy  Revenue  Allocation  for  CCOP  C  &  D,  and  IT  Fixed 

Infrastructure  Contribution. 
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d.  Step  Six:  Develop  the  Value  Equation  Denominator  by 
Assigning  Costs  to  Subprocesses 

(1)  Description:  Costs  are  assigned  directly  to  each  subprocess 
based  on  the  assets  producing  outputs  in  each.  The  cost  of  human  and  IT  assets  that  are 
assigned  to  multiple  processes  are  divided  evenly  throughout  those  subprocesses.  The 
cost  of  human  and  IT  assets  are  summed  in  each  subprocess  to  yield  the  total  cost  per 
subprocess  (CP). 


Subprocess  Name 

Proxy 
Revenue 
Assigned 
to  Sub¬ 
process 
($US) 

Cost 

Assigned 
to  Sub¬ 
process 
($US) 

Proxy 
Revenue 
Assigned 
to  Human 
K($US) 

Cost 

Assigned 
to  Human 

K 

($US) 

P4 

Search/Collection 

$  219,140 

$  64,642 

$  89,980 

$37,276.47 

P8 

Format  Data  for 
Report 

Generation 

$  41,605 

$  52,252 

$  13,868 

$32,326.50 

$  893,000 

$  622,607 

$  269,532 

$  292,533 

Table  20.  P4  and  P8  Total  Cost  Allocation  &  Human  Cost  Allocation. 


Subprocess  Name 

Proxy 
Revenue 
Assigned 
to  CCOP  A 
Process  K 
($US) 

Cost 

Assigned 
to  CCOP 

A 

Process 

K 

($US) 

Proxy 
Revenue 
Assigned 
to  CCOP  B 
Process  K 
($US) 

Cost 

Assigned 
to  CCOP  B 
Process  K 
($US) 

P4 

Search/ 

Collection 

$  89,980 

$  9,675 

$  11,023 

$  7,441 

P8 

Format  Data  for 
Report 

Generation 

$  13,868 

$  9,675 

$  - 

Table  21.  P4  an 

$  269,532 

$  96,750 

$  34,110 

$  22,322 

d  P8  Cost  Allocation  for  CCOP  A  & 

B. 
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Proxy 
Revenue 
Assigned 
to  CCOP 

C  Process 
K 

($US) 

Cost 

Assigned 
to  CCOP 

C 

Process 

K 

($US) 

Proxy 
Revenue 
Assigned 
to  CCOP  D 
Process  K 
($US) 

Cost 

Assigned 
to  CCOP 

D 

Process 

K 

($US) 

Proxy 
Revenue 
Assigned 
to  Fixed 
Infras 
Process 

K 

($US) 

Cost 

Assigned 
to  Fixed 
Infras 
Process 

K 

($US) 

$- 

$- 

$28,156 

$  10,250 

$- 

$- 

$  13,868 

$  10,250 

$  58,253 

$  12,000 

$  19,906 

$  63,462 

$241,667 

$  102,500 

Table  22.  P4  and  P8  Cost  Allocation  for  CCOP  C,  D,  and  Fixed  IT  Infrastructure. 

e.  Steps  Seven,  Eight  and  Nine:  Calculate  the  Value  Equation 
(ROI) 

(1)  Definitions: 

ROK  is  the  Return  on  Knowledge,  a  productivity  ratio 
ROKA  is  the  Return  on  Knowledge  Assets,  a  profitability  ratio 
ROKI  is  the  Return  on  Knowledge  Investment,  the  value  equation 

(2)  Formulas: 

Total  Return  on  Knowledge:  ROK  =  Revenue  /  Cost 

Subprocess  ROK  (as  percentage):  ROKP  =  (RP  /  CP)  x  100% 

Subprocess  ROKA:  ROKAp  =  (  Rp—  Cp  )  /  (  %KP  x  Rtot  ) 

Subprocess  ROKI:  ROKIP  =  (  RP-  CP  )  /  (  CP  ) 


(2)  Description:  The  revenues  and  costs  assigned  to  subprocesses, 
people  and  IT  are  used  to  calculate  the  value  equations. 


KVA  Metrics 

or  Total  K 

Subprocess  Name 

ROK  as  Ratio 

ROK  as  % 

ROKA 

ROKI 

P4 

Search/Collection 

3.39 

339.01% 

70.50% 

239.01% 

P8 

Format  Data  for 

Report  Generation 

0.80 

79.63% 

-25.59% 

-20.37% 

Metrics  for  Aggregated 

14.10 

1410.20% 

157.31% 

410.20% 

Table  23.  P4  and  P8 

CVA  Metrics. 
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Note:  For  Fluman  and  IT  ROK,  ROKA,  and  ROKI,  the  Cost  and  Revenue  of  each  asset 
is  substituted  for  subprocess  cost  and  revenues  in  the  value  equations. 


4.  KVA  Results  for  USS  REDINESS  ICP  with  Actual  Case  Data  Iputs 


Historical  KVA  for  USS  READINESS  for  Intelligence  Collection  Process 

Total  K  Contribution  and  Human  K 


KVA  Metrics  for  Total  K 


KVA  Metrics  for  Human  K 


Subprocess  Name 

ROK  as 
Ratio 

ROK  as  % 

ROKA 

ROKI 

Snbpiocess  Name 

ROK  as 
Ratio 

ROK  as  % 

ROKA 

ROKI 

PI 

Receive  Review 
Request  Tasking 

1.22 

122.11% 

18.10% 

22.11% 

pi 

Receive/Review 
Request  Tasking 

0.71 

70.50% 

-41 .84% 

-29.50% 

P2 

Dele  inline  0  p  Equip 
Mix 

1.21 

120.89% 

17.28% 

20.89% 

P2 

Dete miine  Op  Equip 
Mix 

0.70 

69.80% 

-43.26% 

-30.20% 

P3 

Load  Search 
Func/Coverage 

Plan 

0.82 

81.56% 

-22.62% 

-18.44% 

P3 

Load  Search 
Func/Coveraqe  Plan 

0.96 

95.89% 

-4.29% 

-4.11% 

P4 

Seal  ch  Collection 

3.39 

339.01% 

70.50% 

239.01  % 

P4 

Search/Collection 

2.41 

241.39% 

58.57% 

141.39% 

PS 

Taiget  Data 
Acquisition/Capture 

1.47 

147.28% 

32.10% 

47.28% 

PS 

Target  Data 
Acquisition  Captuie 

0.75 

75.31% 

-32.79% 

-24.69% 

P6 

Taiget  Data 
Piocessinq 

1.37 

136.67% 

26.83% 

36.67% 

P€ 

Target  Data 
Piocessinq 

0.83 

82.90% 

-20.63% 

-17.10% 

P7 

Taiget  Data 

Analysis 

1.21 

121.25% 

17.52% 

21 .25% 

P7 

Target  Data  Analysis 

0.63 

62.75% 

-59.35% 

-37.25% 

P9 

Fonnat  Data  for 
Report  Geneiation 

0.80 

79.63% 

-25.59% 

-20.37% 

P9 

Format  Data  foi 
Repoit  Geneiation 

0.43 

42.90% 

-133.09% 

-57.10% 

P9 

OC  Report 

1.79 

179.19% 

44.19% 

79.19% 

P9 

OC  Report 

0.98 

97.82% 

-2.22% 

-2.18% 

P10 

Tiansmit  Report 

0.83 

82.63% 

-21 .02% 

-17.37% 

P10 

Tiansmit  Report 

0.45 

44.52% 

-124.62% 

-55,48% 

Metiicsfoi  A(|<| legated 

14.10 

1410.20% 

157.31% 

410.20% 

Metiicsfoi  Aggregate 

8.84 

883.79% 

-403.52% 

-116.21% 

Please  note  that  the  flooi  foi  ROKA  is  -100' .  |e.g.,  zeio  t etui n  on  knowledge  ass)  Please  note  that  the  flooi  lot  ROKA  is  100  |e.g„  zero  letuin  on  knowledge  assets) 


KVA  Metrics  for  CCOP  A  K 


Sub-Process 

Name 

ROK  as  Ratio 

ROK  as  % 

ROKA 

ROKI 

Pi 

Receive/ 

Review 

Request' 

Tasking 

1.69 

168.54% 

40.67% 

68.54% 

P2 

Determine 

Op  Equip  Mix 

1.67 

166.86% 

40.07% 

66.86% 

P3 

Load  Search 
Func/ 

Coverage 

Plan 

1.53 

152.91% 

34.60% 

52.91  % 

P4 

Search/ 

Collection 

9.30 

930.03% 

89.25% 

830.03% 

PS 

Target  Data 
Acquisition/ 
Capture 

2.90 

290.15% 

65.54% 

190.15% 

P6 

Taiget  Data 
Piocessinq 

3.19 

319.39% 

68.69% 

219.39% 

P7 

Target  Data 
Analysis 

1.50 

149.98% 

33.33% 

49.98% 

Pd 

Format  Data 
for  Repoit 
Geneiation 

1.43 

143.34% 

30.24% 

43.34% 

P9 

OC  Repoit 

3.16 

315.88% 

68.34% 

215.88% 

pro 

Transmit 

Report 

1.49 

148.75% 

32.77% 

48.75% 

Metiicsfoi  Aqqieqated 

27.86 

2785.86% 

503.50% 

1785.86% 

Please  note  that  the  floor  for  ROKA  is  -100%  (e.g.,  zero  return  on  knowledge  assets) 
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KVA  Metrics  for  CCOP  B  K 

KVA  Metrics  for  CCOP  C  K 

Sub-Process 

Name 

ROK  as 
Ratio 

ROK  as  % 

ROKA 

ROKI 

Sub-Process 

Name 

ROK  as 
Ratio 

ROK  as  % 

ROKA 

ROKI 

PI 

Receive/ 

Review 

Request' 

Taskinq 

PI 

Receive/ 

Review 

Request' 

Tasking 

P2 

Determine 

Op  .'Equip  Mix 

P2 

Determine 
Op/Equip  Mix 

P3 

Load  Search 

Func/ 

Coverage 

Plan 

P3 

Load  Search 

Func/ 

Coverage 

Plan 

P4 

Search/ 

Collection 

1.48 

148.15% 

32.50% 

48.15% 

P4 

Search/ 

Collection 

P5 

Target  Data 

Acquisition/Ca 

ptuie 

1.48 

147.71% 

32.30% 

47.71  % 

P5 

Target  Data 

Acquisition/Ca 

pture 

P6 

Target  Data 
Processing 

1.63 

162.59% 

38.50% 

62.59% 

P6 

Target  Data 
Processing 

4.36 

436.13% 

77.07% 

336.13% 

P7 

Target  Data 
Analysis 

P7 

Target  Data 
Analysis 

5.35 

534.76% 

81 .30% 

434.76% 

P9 

Format  Data 
for  Report 
Generation 

P» 

Format  Data 
for  Report 
Generation 

P9 

OC  Repoit 

P9 

OC  Report 

P10 

Transmit 

Report 

P10 

Transmit 

Report 

Metiicsfoi  Aqqieqated 

4.58 

458.44% 

103.29% 

158.44% 

Metrics  for  Aggregated 

9.71 

970.88% 

158.37% 

770.88% 

Please  note  that  the  floor  for  ROKA  is  -100%  (e.g..  zero  return  on  knowledge  assets) 


KVA  Metrics  for  CCOP  D  K 

KVA  Metrics  for  Fixed  Infrastructure  K 

Sub-Process 

Name 

ROK  as  Ratio 

ROK  as  % 

ROKA 

ROKI 

Sub-Process 

Name 

ROK  as 
Ratio 

ROK  as  % 

ROKA 

ROKI 

PI 

Receive 

Review 

Request 

Tasking 

PI 

Receive 

Review 

Request 

Tasking 

1.95 

194.72% 

48.64% 

94.72% 

P2 

Determine 

Op  Equip  Mix 

P2 

Determine 

Op  Equip  Mix 

1.93 

192.78% 

48.13% 

92.78% 

P3 

Load  Search 
Func 

Coverage 

Plan 

P3 

Load  Search 
Func 

Coverage 

Plan 

1.77 

176.65% 

43.39% 

76.65% 

P4 

Search 

Collection 

P4 

Search 

Collection 

2.75 

274.70% 

63.60% 

174.70% 

P5 

Target  Data 
Acquisition/ 
Capture 

P5 

Taiget  Data 
Acquisition 
Capture 

2.74 

273.88% 

63.49% 

173.88% 

P6 

Target  Data 
Processing 

0.28 

28.18% 

-254.86% 

-71.82% 

P6 

Taiget  Data 
Piocessinq 

3.01 

301.48% 

66.83% 

201.48% 

P7 

Target  Data 
Analysis 

0.35 

34.55% 

-189.41% 

-65.45% 

P7 

Target  Data 
Analysis 

3.70 

369.66% 

72.95% 

269.66% 

P9 

Format  Data 
foi  Repoit 
Geneiation 

P9 

Format  Data 
foi  Repoit 
Geneiation 

1.35 

135.30% 

26.09% 

35.30% 

P9 

OC  Report 

P9 

OC  Repoit 

2.98 

298.16% 

66.46% 

198.16% 

P10 

Transmit 

Repoit 

P10 

Tiansmit 

Repoit 

1.40 

140.41% 

28.78% 

40.41% 

Metiicsfoi  Aggregated 

0.63 

62.73% 

-444.28% 

-137.27% 

Metiicsfoi  Aggiegated 

23.58 

2357.73% 

528.36% 

1357.73% 

Table  24.  KVA  Results  for  USS  READINESS 


E.  ANALYZING  THE  KVA  RESULTS 

KVA  provides  a  unique  and  insightful  means  of  measuring  the  contributions  of 
Information  Technology  and  human  assets  to  the  production  of  process  outputs.  By 
analyzing  an  organization,  process  or  function  at  the  subprocess-level,  this  methodology 
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allows  military  leaders  and  program  managers  to  structure  a  problem  and  solution  set  in 
an  empirical  manner.  The  results  can  then  be  used  to  conduct  comparative  analyses  of 
multiple  human  and  IT  configurations  of  similar  processes  with  the  final  goal  of  ensuring 
the  best  possible  posturing  of  people,  equipment  and  the  increasingly  limited  funding- 
dollars  within  the  organization  and  the  Department  of  Defense. 

While  profitability  is  not  the  goal  of  the  DoD,  and  other  non-profit  organizations, 
it  is  their  goal  and  responsibility  to  the  individual  taxpayer,  to  ensure  that  the  limited 
resources  are  managed  in  a  reasonable,  accountable  and  transparent  manner.  In  this  case, 
KVA  was  used  to  quantify  the  value  added  by  Cryptologic  Carry-On  Program  systems, 
Information  Warfare/Cryptologic  operators,  and  the  enabling  shipbome  infrastructure 
with  which  they  interact.  The  result  is  an  assessment  of  the  efficiency  (productivity)  and 
effectiveness  (profitability)  of  human-system  performance  within  the  Intelligence 
Collection  Process.  To  obtain  a  more  comprehensive  picture  of  CCOP  system 
contribution,  multiple  iterations  of  this  analysis  would  have  to  be  run  across  the  Navy¬ 
wide  enterprise  of  intelligence  collection  platforms.  That  is  not  to  say  though,  that 
analyzing  a  single  process  yields  no  value.  Looking  at  the  USS  READINESS  case  study 
on  its  own,  it  is  possible  to  interpret  the  KVA  analysis  results  and  identify  targets  of 
transformation  within  the  subprocesses  of  the  Intelligence  Collection  Process.  It  is 
possible  to  look  at  the  individual  assets  and  ask  “why?”  an  asset  contributes  more  or  less 
than  others,  and  “how?”  the  perfonnance  of  assets  impacts  the  overall  output  of  the 
process. 

1.  Case  Example  Results 

Looking  back  at  the  two  subprocesses  detailed  in  the  case  study,  P4  and  P8,  a  few 
questions  can  be  asked: 

•  What  makes  P4  such  a  high  performer,  with  an  overall  return  of  239% 
versus  a  return  of  -20.37%  for  P8? 

•  P8,  Format  Data  for  Report  Generation,  only  executes  once  per  KL  report 
(process  output),  yet  almost  a  third  of  all  the  operators  (ComOp  1, 
ComOp2,  ComOp3)  in  this  particular  case  are  assigned  to  this  subprocess 
for  almost  a  fifth  of  the  total  human  cost. 

•  P8  is  more  automated  than  P4.  Could  P8  be  further  automated  or  could  it 
be  performed  by  the  other  operators  in  lieu  of  the  three  communications 
operators  to  yield  higher  efficiency  and  effectiveness? 
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•  P4,  Search  and  Collect,  is  a  knowledge-intensive  function  requiring  a  high 
system  and  human  capital  asset  investment.  Each  process  output  requires 
many  executions  of  P4.  Could  an  even  higher  return  be  achieved  with 
automated  search  and  collection  systems  or  more  operators? 

Other  observations  of  this  case  example  could  include: 

•  Should  the  amount  of  knowledge  in  humans  and  IT  be  adjusted? 

•  Is  a  broader  range  of  training  required  to  allow  all  operators  to  perform 
more  functions? 

•  CCOP  D  is  a  cost-heavy  system  but  executes  very  few  times, 
comparatively,  throughout  the  sample  period.  Is  CCOP  D  appropriate  for 
this  platform  and  mission?  Is  there  a  cheaper  alternative  to  CCOP  D?  Are 
all  the  operators  appropriately  trained  in  the  use  of  CCOP  D? 

These  are  the  kinds  of  questions  CCOP  program  mangers  and  other  stake -holders 
can  begin  to  ask.  The  answers  to  these  questions  can  help  CCOP  program  managers 
allocate  funds  to  new  systems  or  to  existing  systems  for  product  improvement,  or  if 
further  study  deems  it  so,  to  cut  a  system  all  together  from  the  portfolio.  The  results  can 
also  be  used  to  tailor  the  manning  and  training  requirements  of  ISR  crews  deploying  with 
CCOP  systems. 

Finally,  KVA  also  provides  the  structured  data  required  to  perform  various 
methods  of  risk  analysis  and  performance  projections  such  as  Real  Options  Analysis. 
The  combination  of  KVA  historical  performance  metrics  and  real  options  analytic  tools 
will  enable  the  CCOP  Program  Office  and  the  U.S.  Navy  to  estimate  and  compare  the 
future  value  added  of  different  mixes  of  humans  and  systems  as  well  as  a  range  of  new 
initiatives  for  the  deployment  and  employment  options  of  both. 


2.  Current  Limitations  of  Knowledge  Value  Added 

This  case  study  revealed  a  few  limitations  to  the  implementation  of  KVA  to  the 
Intelligence  Collection  Process  as  modeled  in  for  the  USS  READINESS,  which  are 
currently  being  addressed.  They  are: 

•  With  the  raw  data  required  for  the  analysis  residing  in  multiple  databases 
of  varying  classification  levels,  data-gathering  mechanisms  that  are  less 
human-intensive  and  more  automated  need  to  be  created  to  extract  the 
required  information. 

•  Although  the  ICP  in  this  case  study  was  developed  through  the  use  of 
subject  matter  experts,  a  standard  description  and  definition  of  each 
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subprocess  should  be  reached  through  a  Intelligence  Community-wide 
consensus  of  process  stake-holders. 

•  A  more  detailed  research  should  be  conducted  to  analyze  the  knowledge 
embedded  in  each  IT  system  to  accurately  capture  the  benefits  resulting 
from  the  execution  of  particular  system  processes. 

•  The  Market  Comparables  approach  to  valuing  the  outputs  of  non-profit 
organizations,  although  used  as  a  rough  baseline  to  monetize  outputs  in 
this  case  study,  requires  a  more  in-depth  look  at  comparable  organizations 
utilizing  similar  processes  to  produce  similar  outputs.  The  creation  of  a 
broad  database  of  such  organizations  is  currently  being  conducted  to 
benchmark  industries  by  functional  groupings  and  products. 

•  Lastly,  to  provide  a  more  powerful  analysis  of  the  ICP,  a  database  of 
comparable  historical  KVA  information  should  be  created  to  benchmark 
future  work  or  to  provide  a  broader  insight  for  current  work. 
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V.  CONCLUSIONS 


Maritime  ISR  will  remain  at  the  core  of  the  Naval  Operational  Doctrine  and  will 
continue  to  be  an  essential  element  in  improving  the  speed  and  effectiveness  of  naval  and 
joint  operations.  Innovations  resulting  from  the  transfonnation  of  the  DoD’s  business 
practices  must  be  refined  to  meet  the  unique  challenges  posed  by  Intelligence, 
Surveillance,  and  Reconnaissance  programs.  It  is  the  responsibility  of  program  managers 
and  DoD  officials  to  do  their  very  best  to  create  realistic  objectives,  clear  guidance  and  an 
effective  organization  when  executing  the  resources  of  the  United  States  in  defense  of  the 
nation.  Dwight  D.  Eisenhower  once  said: 

Every  gun  that  is  made,  every  warship  launched,  every  rocket  fired, 
signifies  in  the  final  sense  a  theft  from  those  who  hunger  and  are  not  fed, 
those  who  are  cold  and  are  not  clothed. 

-Dwight  D.  Eisenhower 

The  citizens  of  the  country  are  entitled  to  see  standards  for  performance  so  they  can  judge 
whether  the  national  objectives  are  being  met  in  a  responsible  manner. 

Congress  recently  criticized  the  DoD’s  acquisition  and  testing  and  evaluation  of 
Intelligence,  Surveillance  and  Reconnaissance  (ISR)  systems,  stating  that  there  has  been 
“inadequate  data  to  compare  systems’  capabilities  and  costs  across  the  spectrum  of 
intelligence  programs,  an  imbalance  between  collection  and  analysis  programs,  and  an 
intelligence  effort  that  does  not  reflect  an  optimal  allocation  of  resources. ”36  Section  355 
of  the  FY2004  Intelligence  Authorization  Act  (P.L.  108-177)  required  a  report  from  the 
Director  of  Central  Intelligence  (DCI)  and  the  Secretary  of  Defense  assessing  progress  in 
the  development  of  “a  comprehensive  and  uniform  analytical  capability  to  assess  the 
utility  and  advisability  of  various  sensor  and  platfonn  architectures  and  capabilities  for 
the  collection  of  intelligence  ...  [and]  the  improvement  of  coordination  between  the 
Department  [of  Defense]  and  the  intelligence  community  on  strategic  and  budgetary 
planning. ”37 

36  Best,  Richard.  Intelligence,  Surveillance,  and  Reconnaissance:  Issues  for  Congress.  CRS  Report 
for  Congress  (RL  32508).  Washington:  Congressional  Research  Service,  22  Feb  2005.  p.  2. 

37  Ibid. 
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Assessing  the  impact  and  overall  value  of  information  technology  investments, 
such  as  ISR  systems,  has  been  a  continuing  challenge,  both  in  business  and  in  the  DoD. 
The  solution  to  the  valuation  of  IT  systems  has  been  referred  to  as  one  of  the  “holy  grails 
of  the  Information  Systems  (IS)  field,”  most  notably,  at  a  May  2005  presentation  of  this 
case  study  at  the  DoD’s  2nd  Annual  Acquisitions  Symposium  in  Monterey,  California.  But 
even  skeptics  of  the  various  methodologies  proposed  for  measuring  the  value  of  IT 
acquisitions  agree  that  a  common  framework  must  be  created  for  understanding, 
evaluating,  and  in  the  end,  justifying  the  impact  of  government  investments  in 
information  technology  on  the  overall  successful  completion  of  the  national  security 
mission  of  the  United  States. 

The  challenges  in  determining  the  value  and  risk  associated  with  ISR  systems 
acquisitions  are  plenty.  In  this  particular  case,  the  use  of  Commercial  of  the  Shelf  and 
Government  off  the  Shelf  (COTS/GOTS)  systems  and  open  systems  architectures  in  ISR 
systems  provide  a  level  of  risk,  which  must  be  managed.  Technological  complexities,  the 
reliance  on  evolving  software  and  systems,  shortened  acquisitions  timelines  and  funding 
instability  all  contribute  to  this  risk  in  Navy  ISR  systems  such  as  the  Cryptologic  Carry- 
On  Program  (CCOP).  Although  the  DoD  has  instituted  rigorous  types  of  testing  and 
evaluation  (T&E)  for  all  of  its  programs  and  projects  to  mitigate  risk,  metrics  for  IT 
systems  have  lacked  the  requisite  depth  for  meaningful  valuation.  Crucial  to  successful 
T&E  is  the  development  of  measurable  Key  Performance  Parameters  (KPPs)  and 
Measures  of  Effectiveness  (MOEs)  to  provide  accurate  projections  of  system 
performance  in  a  variety  of  operational  environments. 

It  was  the  goal  of  this  research  to  provide  the  means  to  extract  measures  of  value 

and  effectiveness  to  the  CCOP  Program  office  through  the  use  of  the  Housel-Kenevsky 

Knowledge  Value  Added  (KVA)  Methodology.  Applying  KVA  to  the  USS 

READINESS  Case  Study  showed  that  the  program  managers  could  build  metrics  that  are 

meaningful  and  useful  in  performing  sound  financial  analysis  of  each  system’s 

performance  at  the  process  and  subprocess  level.  KVA  analysis  also  identified  a  new 

category  and  source  of  raw  data  which  can  provide  insights  into  the  relationship  of  cost 

and  value  of  organizations,  processes,  and  asset  investments.  This  new  data  allows 

managers  and  senior  decision  makers  to  discuss  the  “value”  of  seemingly  intangible 
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assets  in  a  defensible,  empirical  and  replicable  manner.  Lastly,  KVA  facilitates  the 
transfonnation  and  continuous  process  improvement  of  the  DoD’s  global  intelligence 
mission.  Through  KVA  analysis,  the  operational  value  of  CCOP  systems  can  be 
measured  and  managed  to  ensure  a  responsible  stewardship  of  the  nation’s  resources  and 
ensure  that  the  soldiers  and  sailors  who  use  these  systems  are  receiving  the  right  tools 
with  the  right  capabilities  required  to  perfonn  their  duties  in  defense  of  the  nation. 
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VI.  RECOMMENDATIONS 


The  quest  for  the  holy  grail  of  information  technology  is  far  from  over.  An 
accurate  valuation  of  Cryptologic  Cary-On  Program  systems  will  come  through  an 
organizational  transformation  and  operational  transformation  as  well. 

The  Knowledge  Value  Added  (KVA)  Methodology,  as  any  other  methodology,  is 
only  as  effective  and  meaningful  as  the  data  used  to  apply  it.  As  indicated  in  the  case 
study,  data  collection  challenges  remain.  It  is  recommended  that  processes,  both  human 
and  automated,  be  created  to  gather  the  required  perfonnance  data  in  near-real-time. 
Automated  logging  of  system  utilization  and  performance  are  readily  available  in  many 
business  applications.  Adapting  such  mechanisms  directly  into  CCOP  systems  would 
facilitate  the  analysis  of  their  perfonnance.  Human  activity  and  perfonnance 
mechanisms  have  also  been  used  in  the  past,  both  in  business  and  in  the  DoD. 

KVA  is  a  robust  methodology,  when  applied  throughout  an  enterprise  with  the 
appropriate  inputs,  it  is  capable  of  analyzing  the  data  and  returning  defensible, 
quantifiable  results  in  near-real-time.  Although  several  accounting  software  packages 
have  included  KVA  analytical  capabilities,  the  research  team  at  the  Naval  Postgraduate 
School  identified  the  GaussSoft  Valuation  Software  as  the  most  capable  software 
platform  for  conducting  the  level  of  analysis  required  by  DoD  program  managers  for 
valuing  IT  systems  and  processes.  A  brief  overview  of  GaussSoft  and  screen  shots  of  the 
multi-faceted  reports  produced  by  the  software  package  are  provided  in  Appendix  B.  It  is 
recommended  that  this  software  package  be  deployed  to  receive  real-time  system  and 
process  inputs  from  deployed  units  operating  CCOP  systems  as  a  proof-of-concept  and 
operational  test  of  the  software  capabilities. 

A  true  measure  of  the  impact  of  CCOP  systems  will  require  that  the  methodology 
and  data  gathering  be  analyzed  and  refined  over  a  larger  sampling  of  time.  KVA,  in  its 
current  form  is  ready  for  deployment;  it  is  recommended  that  a  study  be  conducted  on 
CCOP  systems  at  the  Carrier  Strike  Group  (CSG)  or  Expeditionary  Strike  Group  (ESG) 
level  over  the  course  of  one  deployment  to  begin  establishing  performance  baselines  for 
systems  and  processes.  It  is  also  recommended  that  a  community-wide  effort  be  enacted 
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along  with  the  system  developers  to  determine  a  universally  accepted  description  of  the 
embedded  knowledge  and  required  learning  time  of  each  system  and  process.  A  more  in 
depth  and  extensive  research  should  be  conducted  on  the  Market  Comparables  Approach 
to  include  a  valuation  study  of  the  intelligence  products  produced  by  private  military 
corporations  (PMCs)  as  well  as  competitive  and  business  intelligence  organizations  to 
achieve  a  common  baseline  which  can  be  applied  directly  to  DoD  intelligence  products. 
A  KVA  baseline  should  also 

Lastly,  it  is  recommended  that  an  organization  be  identified  to  maintain  KVA 
databases  for  CCOP  systems.  Such  organization  could  be  a  central  repository  for  system 
performance  data  as  well  as  a  ready  source  of  performance  and  valuation  reports  which 
can  be  tapped  quarterly  or  yearly  by  program  managers  wishing  to  make  informed 
acquisition  decisions. 
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APPENDIX  A.  USS  READINESS  KVA  ANALYSIS 


Historical  Learning  Time  and  Automation  Data  -  6  Month  Deployment  Sample  Period 
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0.00 

0.00 

0.00 

36.00 

363223.92 

36.00 

399828.08 

0.00 

0.00 

0.00 

0.00 

0.00 

0.00 

0.00 

0.00 

0.00 

0.00 

0.00 

0.00 

0.00 

0.00 

78.00 

388792.06 

0.00 

0.00 

78.00 

403466.98 

0.00 

0.00 

0.00 

0.00 

0.00 

0.00 

78.00 

388792.06 

0.00 

0.00 

78.00 

403466.98 

0.00 

0.00 

0.00 

0.00 

0.00 

0.00 

79.00 

393776.58 

0.00 

0.00 

79.00 

408639.64 

235.00 

2371045.03 

235.00 

2609988.83 

90.00 

1225628.77 

235.00 

1171360.71 

235.00 

2581310.50 

235.00 

1215573.61 

92.00 

928238.91 

92.00 

1021782.86 

0.00 

0.00 

0.00 

0.00 

0.00 

0.00 

0.00 

0.00 

0.00 

0.00 

199.00 

2210160.76 

199.00 

2710001 .38 

0.00 

0.00 

0.00 

0.00 

0.00 

0.00 

0.00 

0.00 

68.00 

755230.81 

68.00 

926030.62 

0.00 

0.00 

0.00 

0.00 

0.00 

0.00 

235.00 

2371045.03 

235.00 

2609988.83 

235.00 

3200252.89 

235.00 

1171360.71 

235.00 

2581310.50 

235.00 

1215573.61 

8041374.01 

11817140.93 

9287542.42 

3514082.12 

7743931 .50 

3646720.82 
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Historical  KVA  for  USS  READINESS  for  Intelligence  Collection  Process 

Total  K  Contribution  and  Human  K 


Assi 
(jne 
<1  to 
Pro 

ces 

ses 

Asset 

Avg  Annual 
Unit  Costs 

Avg  Annual 
Unit  Bonus 

Budget 
(Cost)  per 
Sample  Pd 

Cost  per 
Operator 
per  Sub¬ 
process 

Proxy  Revenue  &  Cost  Assumptions 

1,2,9 

Div  Officer 

$  79,298 

i 

$  39,649 

$13,216.33 

Market  Comparable  Price  Per  Unit  (avg) 

$  3.800 

1,2,9 

Div  LPO 

$  79,298 

$ 

$  39,649 

$9,912.25 

Avg=  Reports  executed  sample  pd 

235 

3-7, £ 

Sit|0|)  1 

$  61 ,883 

$  35,475 

$  48,679 

$8,113.17 

Avg  Proxy  for  Revs  -  Sample  Pd  = 

$  893.000 

3-7 

Si(|0|>  2 

$  58,190 

$  14,958 

$  36,574 

$7,314.80 

Avg  cost  for  IT  Fixed  Infrastructure  (annual)  = 

$  205,000 

4-7 

Sit|0|>  3 

$  44,391 

$  17,177 

$  30,784 

$7,696.00 

All  other  fixed  costs  (annual)  = 

$ 

4-6 

Si(|0p  4 

$  43,397 

$ 

$  21 ,699 

$7,232.83 

4-6 

SitjOp  5 

$  41,518 

i 

$  20,759 

$6,919.67 

Length  of  Sample  Pd  as  %  of  Year  = 

50.00% 

8,10 

ComOpI 

$  44,391 

$  22,196 

$1 1 ,097.75 

8.10 

ComOp2 

$  43,397 

$  21 ,699 

$10,849.25 

i 

8,10 

ComOp3 

$  41,518 

$  20,759 

$10,379.50 

Total  Human 

$  302,446 

ill -io 

CCOP  A 

$  193,500 

$  96,750 

4-6 

CCOP  B 

$  44,643 

$  22,322 

6,7 

CCOP  C 

$  24,000 

$  12,000 

6,7 

CCOP  D 

$  126,923 

$  63,462 

Total  IT 

*  194,533 

1-10 

Fixed  IT  Infrastructure 

$  102,500 

1-10 

Other  Fixed  Costs 

GRAND  TOTALS 

$  599.479 

Historical  KVA  for  USS  READINESS  for  Intelligence  Collection  Process 

Total  K  Contribution  and  Human  K 


Subprocess  Name 

K  for  IT 

4  automation  & 
inf  r  as) 

Kfot  Humans 

Total  K 

%  of  Total 

K  per  sub¬ 
process 

Proxy 
Revenue 
Assigned 
to  Sub¬ 
process 
($US) 

Cost 

Assigned 
to  Sub- 
process 
($US| 

%  of 
Total  K 
for 

Human 
pet  Sub¬ 
process 

Proxy 
Revenue 
Assigned  to 
Human  K 
($US) 

Cost 

Assigned  to 
Human  K 
<$US) 

Pi 

Receive  Review 
ReguestTasking 

3,062,999.10 

1 ,377,273.60 

4,440,272.70 

5.8870% 

$  52,571 

$ 

43,054 

1 .8260% 

$ 

16,306 

$23,128.58 

P2 

Determine  Op  Eguip 
Mix 

3,032,519.84 

1 ,363,568.64 

4,396,088.48 

5.8285% 

$  52,048 

$ 

43,054 

1 .8079% 

$ 

16,144 

$23,128.58 

P3 

Load  Search 
Func/Coverage 

Plan 

2,778,886.96 

1 ,249,522.94 

4,028,409.91 

5.3410% 

$  47,695 

$ 

58,482 

1 .6567% 

$ 

14,794 

$15,427.97 

P4 

Search  Collection 

10,909,092.30 

7,599,933.50 

18,509,025.80 

24.5398% 

$  219,140 

$ 

64,642 

10.0762% 

$ 

89,980 

$37,276.47 

P5 

Target  Data 
Acguisition  Capture 

5,670,328.97 

2,371 ,045.03 

8,041 ,374.01 

10.6615% 

$  95,207 

$ 

64,642 

3.1436% 

$ 

28,072 

$37,276.47 

P6 

Target  Data 
Processing 

9,207,152.10 

2,609,988.83 

11,817,140.93 

15.6675% 

$  139,911 

$ 

102,373 

3.4604% 

$ 

30,901 

$37,276.47 

P7 

Target  Data 

Analysis 

8,061 ,913.65 

1 ,225,628.77 

9,287,542.42 

12.3137% 

$  109,961 

$ 

90,692 

1 .6250% 

$ 

14,511 

$23,123.97 

Pd 

Format  Data  for 
Report  Generation 

2,342,721 .41 

1,171 ,360.71 

3,514,082.12 

4.6591% 

$  41,605 

$ 

52,252 

1 .5530% 

$ 

13,868 

$32,326.50 

P9 

OC  Report 

5,162,621 .00 

2,581 ,310.50 

7,743,931 .50 

10.2671% 

$  91,685 

$ 

51 ,167 

3.4224% 

$ 

30,562 

$31 ,241 .75 

P10 

Transmit  Report 

2,431 ,147.21 

1,215,573.61 

3,646,720.82 

4.8349% 

$  43,176 

$ 

52,252 

1.6116% 

$ 

14,392 

$32,326.50 

52,659,382.55 

22,765,206.13 

75,424,588.68 

100.0000% 

$  893,000 

$ 

622,607 

30.1827% 

$ 

269,532 

$  292,533 
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Historical  KVA  for  USS  READINESS  for  Intelligence  Collection  Process 

Total  K  Contribution  and  Human  K 


KVA  Metrics  for  Total  K 


KVA  Metrics  for  Human  K 


Suhprocess  Name 

ROK  as 
Ratio 

ROK  as  % 

ROKA 

ROKI 

Suhprocess  Name 

ROK  as 
Ratio 

ROK  as  % 

ROKA 

ROKI 

PI 

Receive  Review 
Reguest  Tasking 

1.22 

122.11% 

18.10% 

22.11% 

Pi 

Receive  Review 
Reguest  Tasking 

0.71 

70.50% 

-41 .84% 

-29.50% 

P2 

Determine  Op /Equip 
Mix 

1.21 

120.89% 

17.28% 

20.89% 

P2 

Determine  Op/Equip 
Mix 

0.70 

69.80% 

-43.26% 

-30.20% 

P3 

Load  Search 
Func/Coverage 

Plan 

0.82 

81 .56% 

-22.62% 

-18.44% 

P3 

Load  Search 
Func/Coverage  Plan 

0.96 

95.89% 

-4.29% 

-4.11% 

P4 

Search/Collection 

3.39 

339.01  % 

70.50% 

239.01% 

P4 

Search/Collection 

2.41 

241 .39% 

58.57% 

141.39% 

P5 

Target  Data 
Acguisition/Captuie 

1.47 

147.28% 

32.10% 

47.28% 

PS 

Target  Data 
Acguisition/Capture 

0.75 

75.31  % 

-32.79% 

-24.69% 

P6 

Target  Data 
Processing 

1.37 

136.67% 

26.83% 

36.67% 

P6 

Target  Data 
Processing 

0.83 

82.90% 

-20.63% 

-17.10% 

PI 

Target  Data 

Analysis 

1.21 

121.25% 

17.52% 

21.25% 

P7 

Taiget  Data  Analysis 

0.63 

62.75% 

-59.35% 

-37.25% 

PS 

Format  Data  for 
Report  Generation 

0.80 

79.63% 

-25.59% 

-20.37% 

PS 

Format  Data  for 
Report  Generation 

0.43 

42.90% 

-133.09% 

-57.10% 

P9 

OC  Report 

1.79 

179.19% 

44.19% 

79.19% 

P9 

OC  Report 

0.98 

97.82% 

-2.22% 

-2.18% 

pie 

Transmit  Report 

0.83 

82.63% 

-21 .02% 

-17.37% 

pie 

Transmit  Report 

0.45 

44.52% 

-124.62% 

-55.48% 

Metrics  for  Aggregated 

14.10 

1410.20% 

157.31% 

410.20% 

Metrics  for  Aggregate 

8.84 

883.79% 

-403.52% 

-116.21% 

Please  note  that  the  floor  for  ROKA  is  -100%  (e.g..  zero  return  on  knowledge  assj  Please  note  that  the  floor  for  ROKA  is  -100%  (e.g..  zero  return  on  knowledge 
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Historical  KVA  for  USS  READINESS  for  Intelligence  Collection  Process 

IT  Contribution 


Pioij  Revenue  &  Cost  Assumptions 

CCOP  A 

K 

CCOP  B 

K 

CCOPC 

K 

CCOP  D 

K 

Fixed  Infras 
K 

Total  IT 

K 

Market  Comparable  Price  Per  Unit  (avg) 

$  3.800 

Pt 

1,377,273.60 

1,685,725.50 

3,062,999.10 

Avgi  Reports  executed/sample  pd 

235 

P2 

1,363,568.64 

1,668,951.20 

3,032,519.84 

Avg  Proi|  for  Revs  -  Sample  Pd  = 

t  893.000 

PS 

1,249,522.34 

1,523,364.02 

2,778,886.36 

Avg  cost  for  IT  Fixed  Infrastructure  (annual)  = 

t  205.000 

P4 

7,593,933.50 

931,017.15 

2,378,141.64 

10,909,092.30 

All  other  fixed  costs  (annual)  = 

* 

P5 

2,371,045.03 

928,238.31 

2,371,045.03 

5,670,328.97 

P6 

2,603,988.83 

1,021,782.86 

2,210,160.76 

755,230.81 

2,603,388.83 

9,207,152.10 

Length  of  Sample  Pd  as  %  of  Year  = 

50.00% 

Pf 

1,225,628.77 

2,710,001.38 

326,030.62 

3,200,252.89 

8,061,913.65 

pa 

1,171,360.71 

1,171,360.71 

2,342,721.41 

PS 

2,581,310.50 

2,581,310.50 

5,162,621.00 

p/a 

1,215,573.61 

1,215,573.61 

2,431,147.21 

22,765,206.13 

2,881,038.32 

4,920,162.14 

1,681,261.43 

20,411,713.93 

52,653,382.55 

Subprocess  Name 

K  for  IT 

(a(toB)tio>  tc 
ikfris) 

K  for 

Humans 

Total  K 

%  of  Total  K 
for  CCOP  A 

Prox| 
Revenue 
Assigned 
to  CCOP 

A  Process 

K 

(*US) 

Cost 
Assigned 
to  CCOP 

A 

Process 

K 

(*US) 

%  of  Total  K 
for  CCOP  B 

Proxg 
Revenue 
Assigned 
to  CCOP 

B  Process 

K 

(*US) 

Cost 
Assigned 
to  CCOP  B 

Process  K 
(*US) 

Pt 

Receive/ 

Review 

Request/ 

Tasking 

3,062,999.10 

1,377,273.60 

4,440,272.70 

1.83% 

t 

16,306 

$ 

9,675 

0.00% 

$ 

P2 

Determine 
Op/Equip  Mix 

3,032,519.84 

1,363,568.64 

4,396,088.48 

1.81% 

J 

16,144 

t 

9,675 

0.00% 

J 

PS 

Load  Search 

Func/ 

Coverage 

2,778,886.96 

1,249,522.94 

4,028,409.91 

1.66% 

t 

14,794 

$ 

9,675 

0.00% 

$ 

P4 

Search/ 

Collection 

10,909,092.30 

7,599,933.50 

18,509,025.80 

10.08% 

t 

89,980 

$ 

9,675 

1.23% 

J 

11,023 

t 

7,441 

P5 

Target  Data 
Acquisition/ 
Capture 

5,670,328.97 

2,371,045.03 

8,041,374.01 

3.14% 

t 

28,072 

t 

9,675 

1.23% 

t 

10,990 

i 

7,441 

pa 

Target  Data 
Processing 

9,207,152.10 

2,609,988.83 

11,817,140.93 

3.46% 

t 

30,901 

J 

9,675 

1.35% 

$ 

12,098 

$ 

7,441 

PT 

Target  Data 
Analgsis 

8,061,913.65 

1,225,628.77 

9,287,542.42 

1.62% 

t 

14,511 

t 

9,675 

0.00% 

t 

PS 

Format  Data 
for  Report 
Generation 

2,342,721.41 

1,171,360.71 

3,514,082.12 

1.55% 

t 

13,868 

$ 

9,675 

0.00% 

$ 

P3 

QC  Report 

5,162,621.00 

2,581,310.50 

7,743,931.50 

3.42% 

t 

30,562 

t 

9,675 

0.00% 

J 

p/a 

Transmit 

Report 

2,431,147.21 

1,215,573.61 

3,646,720.82 

1.61% 

t 

14,392 

$ 

9,675 

0.00% 

$ 

52,659,382.55 

22,765,206.13 

75,424,588.68 

30.18% 

t 

269,532 

t 

96,750 

3.82% 

i 

34,110 

t 

22,322 
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Suhprocess  Name 

%  of  Total  K 
for  CCOP  C 

Proxy 
Revenue 
Assigned  to 
CCOP  C 
Process  K 
($US> 

Cost 

Assigned  to 
CCOP  C 
Piocess  K 
($US) 

%  of  Total  K 
for  CCOP  D 

Proxy 
Revenue 
Assigned  to 
CCOP  D 
Process  K 
($US) 

Cost 

Assigned 
to  CCOP 

D 

Process 

K 

($US) 

%  of  Total 
K  for 
Fixed  IT 
Infrastruct 

ure 

Proxy 
Revenue 
Assigned 
to  Fixed 
Infras 
Process  K 
($US) 

Cost 

Assigned  to 
Fixed 
Infras 
Process  K 
($US) 

Pi 

Receive 

Review 

Request 

Tasking 

$ 

$ 

0.022350 

$ 

19,958 

$ 

10,250 

P2 

Determine 

Op  Equip  Mix 

$ 

$ 

0.022127 

$ 

19,760 

$ 

10,250 

P3 

Load  Search 
Func/ 

Coverage 

$ 

$ 

0.020277 

$ 

18,107 

$ 

10,250 

P4 

Search/ 

Collection 

0.00% 

$ 

0.00% 

$ 

3.15% 

$ 

28,156 

$ 

10,250 

P5 

Target  Data 
Acquisition 
Capture 

0.00% 

$ 

0.00% 

f 

3.14% 

$ 

28,072 

$ 

10,250 

P6 

Target  Data 
Processing 

2.93% 

$ 

26,168 

$  6,000 

1.00% 

$ 

8,942 

$  31 ,731 

3.46% 

$ 

30,901 

$ 

10,250 

P7 

Target  Data 
Analysis 

3.59% 

$ 

32,085 

$  6,000 

1.23% 

$ 

10,964 

$  31 ,731 

4.24% 

$ 

37,890 

$ 

10,250 

P9 

Foimat  Data 
foi  Repoit 
Generation 

0.00% 

$ 

0.00% 

$ 

1.55% 

$ 

13,868 

$ 

10,250 

P9 

OC  Repoit 

0.00% 

$ 

0.00% 

$ 

3.42% 

$ 

30,562 

$ 

10,250 

P10 

Transmit 

Report 

0.00% 

$ 

0.00% 

$ 

1.61% 

$ 

14,392 

$ 

10,250 

6.52% 

$ 

58,253 

$  12,000 

2.23% 

$ 

19,906 

$  63,462 

27.06% 

$ 

241 ,667 

$ 

102,500 

KVA  Metrics  for  CCOP  A  K 


Sub -Process 

Name 

ROK  as  Ratio 

ROK  as  % 

ROKA 

ROKI 

PI 

Receive 

Review 

Request 

Tasking 

1.69 

168.54% 

40.67% 

68.54% 

P2 

Determine 

Op  Equip  Mix 

1.67 

166.86% 

40.07% 

66.86% 

P3 

Load  Search 
Func/ 

Coverage 

Plan 

1.53 

152.91% 

34.60% 

52.91% 

P4 

Search 

Collection 

9.30 

930.03% 

89.25% 

830.03% 

P5 

Target  Data 
Acquisition 
Capture 

2.90 

290.15% 

65.54% 

190.15% 

P6 

Target  Data 
Processing 

3.19 

319.39% 

68.69% 

219.39% 

P7 

Target  Data 
Analysis 

1.50 

149.98% 

33.33% 

49.98% 

P9 

Format  Data 
for  Repoit 
Generation 

1.43 

143.34% 

30.24% 

43.34% 

P9 

OC  Report 

3.16 

315.88% 

68.34% 

215.88% 

P1« 

Transmit 

Report 

1.49 

148.75% 

32.77% 

48.75% 

Metrics  foi  Aggregated 

27.86 

2785.86% 

503.50% 

1785.86% 

Please  note  that  the  floor  foi  ROKA  is  -100%  (e.<j..  zero  return  on  knowledge  assets) 
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KVA  Metrics  for  CCOP  B  K 

KVA  Metrics  for  CCOP  C  K 

Sub-Process 

Name 

ROK  as 
Ratio 

ROK  as  % 

ROKA 

ROKI 

Sub-Process 

Name 

ROK  as 
Ratio 

ROK  as  % 

ROKA 

ROKI 

PI 

Receive/ 

Review 

Request' 

Taskinq 

PI 

Receive/ 

Review 

Request' 

Tasking 

P2 

Determine 

Op  Equip  Mix 

P2 

Determine 
Op/Equip  Mix 

P3 

Load  Search 

Func/ 

Coverage 

Plan 

P3 

Load  Search 

Func/ 

Coverage 

Plan 

P4 

Search/ 

Collection 

1.48 

148.15% 

32.50% 

48.15% 

P4 

Search/ 

Collection 

P5 

Target  Data 
Acquisition'  Ca 
ptuie 

1.48 

147.71% 

32.30% 

47.71  % 

P5 

Target  Data 

Acquisition/Ca 

ptuie 

P6 

Target  Data 
Processing 

1.63 

162.59% 

38.50% 

62.59% 

P6 

Target  Data 
Processing 

4.36 

436.13% 

77.07% 

336.13% 

P7 

Target  Data 
Analysis 

P7 

Target  Data 
Analysis 

5.35 

534.76% 

81 .30% 

434.76% 

P9 

Format  Data 
for  Report 
Generation 

P» 

Format  Data 
for  Report 
Generation 

P9 

OC  Repoit 

P9 

OC  Repoit 

P10 

Transmit 

Report 

P10 

Transmit 

Report 

Metiicsfoi  Aggregated 

4.58 

458.44% 

103.29% 

158.44% 

Metrics  foi  Aggregated 

9.71 

970.88% 

158.37% 

770.88% 

Please  note  that  the  floor  for  ROKA  is  -100%  (e.g.,  zero  return  on  knowledge  assets) 


KVA  Metrics  for  CCOP  D  K 

KVA  Metrics  for  Fixed  Infrastructure  K 

Sub-Process 

Name 

ROK  as  Ratio 

ROK  as  % 

ROKA 

ROKI 

Sub-Process 

Name 

ROK  as 
Ratio 

ROK  as  % 

ROKA 

ROKI 

PI 

Receive 

Review 

Request 

Tasking 

PI 

Receive 

Review 

Request 

Taskinq 

1.95 

194.72% 

48.64% 

94.72% 

P2 

Determine 

Op  Equip  Mix 

P2 

Determine 

Op  Equip  Mix 

1.93 

192.78% 

48.13% 

92.78% 

P3 

Load  Search 
Func 

Coverage 

Plan 

P3 

Load  Search 
Func 

Coverage 

Plan 

1.77 

176.65% 

43.39% 

76.65% 

P4 

Search 

Collection 

P4 

Search 

Collection 

2.75 

274.70% 

63.60% 

174.70% 

P5 

Target  Data 
Acquisition/ 
Capture 

P5 

Taiget  Data 
Acquisition 
Capture 

2.74 

273.88% 

63.49% 

173.88% 

P6 

Target  Data 
Processing 

0.28 

28.18% 

-254.86% 

-71.82% 

P6 

Taiget  Data 
Piocessinq 

3.01 

301.48% 

66.83% 

201.48% 

P7 

Target  Data 
Analysis 

0.35 

34.55% 

-189.41% 

-65.45% 

P7 

Target  Data 
Analysis 

3.70 

369.66% 

72.95% 

269.66% 

P9 

Format  Data 
foi  Repoit 
Geneiation 

P9 

Format  Data 
foi  Repoit 
Geneiation 

1.35 

135.30% 

26.09% 

35.30% 

P9 

OC  Report 

P9 

OC  Repoit 

2.98 

298.16% 

66.46% 

198.16% 

P10 

Transmit 

Repoit 

P10 

Tiansmit 

Repoit 

1.40 

140.41% 

28.78% 

40.41% 

Metiicsfoi  Aggregated 

0.63 

62.73% 

-444.28% 

-137.27% 

Metiicsfoi  Aqqieqated 

23.58 

2357.73% 

528.36% 

1357.73% 
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APPENDIX  B.  GAUSSSOFT  OVERVIEW 


[GAUSS  Overview  provided  courtesy  of  GaussSoft,  Inc.  <http:www.gaussoft.com>] 

GAUSS  is  a  line  of  software  created  by  GaussSoft,  Inc.,  a  privately  held  US 
corporation  founded  in  1993,  with  headquarters  in  San  Jose,  California  and  an  extended 
presence  with  offices  and  partners  in  NorthAmerica,  Europe  and  Latin  America. 

GaussSoft  delivers  scalable  Enterprise  Performance  Intelligence  solutions. 
GaussSoft  products  enable  large  and  medium-sized  companies  to  control  and  reduce  the 
cost  of  enterprise  operations,  increase  profitability  and  improve  organizational 
productivity  by  providing  flexibility,  scalability  and  ease  of  use. 

GaussSoft’ s  solutions  are  built  on  an  integrated  suite  of  high  performance 
products  for  Profit  and  Cost  Analysis,  Multidimensional  Query,  and  Activity  Reporting 
that  are  scalable,  function-rich,  and  easy  to  use. 

GaussSoft  has  installed  performance  intelligence  solutions  in  over  200  enterprise 
and  consulting  companies  all  around  the  world,  including  telecommunication,  banking, 
manufacturing  and  agribusiness  firms  and  government  organizations.  They  have  been 
implemented  in  customer  premises  by  leading  consulting  firms  including  Deloitte, 
KPMG  and  Price. 

GaussSoft  suite  includes: 

Gauss  -  Profit  and  Cost  Allocation  Engine:  This  strategic  decision-making  and 
analysis  solution  enables  companies  to  know  which  products,  services,  and  customers  are 
making  profits  and  which  aren’t.  Using  different  value  and  costing  methodologies  this 
solution  helps  reduce  and  control  the  cost  of  enterprise  operations,  increase  profitability 
and  improve  organizational  productivity. 

Gauss  -  KVA:  Knowledge  Value  Added  (KVA)  is  a  methodology  that  allows  any 
organization  to  calculate  the  economic  performance  of  core  processes  by  providing  an 
objective  way  to  allocate  revenue  to  the  processes  at  any  level  within  the  organization. 
Knowing  how  much  revenue  corporate  knowledge  is  producing,  allows  organizations  to 
dramatically  improve  their  effectiveness  and  efficiency. 
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Gauss  -  Planning:  This  enterprise  collaborative  solution  allows  thousands  of 
users  to  perform  corporate  enterprise  planning,  including  financial  planning,  budgeting 
and  forecasting  up  to  10  times  faster.  When  used  with  Gauss  Profit  and  Gauss  KVA,  an 
organization  can  create  plans  optimized  for  profitability  and  value. 

Gauss  -  Radial  Viewer:  This  is  a  Business  Intelligence  (BI)  front-end  with 
graphical  interaction.  This  tool  enables  all  End  Users  to  create  their  own  queries  and 
professional  looking  reports  from  scratch  -in  seconds-. 

Figures  5-7  are  graphical  outputs  of  GaussSoft  products. 


Dominant4 


i  x  a h <® 

Dominants 


Profit  and  Loss  (Account  detail)  /  SP01  SP01  /  Lines:  A 


Total:  108,707.54  No.Units:  24.00  Cost  p.  unit:  4,529.48  Terminal:  T 


Figure  5.  GaussSoft  Accumulator  View  for  KVA  Case  Study. 
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Figure  6. 


GaussSoft  Radial  Viewer  Report  Design  Screen. 


[3c]  Widths  1 0  Auto  jr] 


Process  type  * 

Process 

KVA 

Revenue 

Cost 

Profit 

ROI 

Human 

SP01  SubprocessOI 

1 

14,961 

108,708 

-93,746 

-86 

SP02  Subprocess02 

1 

7,481 

32.489 

-25,009 

-77 

SP03  SP03 

2 

26,182 

56,866 

-30,684 

-54 

SP04  SP04 

52 

589,093 

40,335 

548,758 

1,360 

Total  Human 

56 

637  717 

238,398 

399,319 

168 

It 

ITP01  PROC  01 

18 

202  598 

300  000 

-97,402 

-32 

ITPA  PROC  A 

23 

264,936 

193,500 

71,436 

37 

ITPB  PROC  B 

4 

46,909 

44,643 

2,266 

5 

ITPC  PROC  C 

3 

29,486 

24,000 

5,486 

23 

ITPD  PROC  D 

6 

7Z374 

126.923 

-54,549 

-43 

Total  It 

54 

616,304 

689,066 

-72  762 

-11 

Not  defined 

SP05  SP05 

9 

102,234 

31,457 

70.777 

225 

SP06  SP06 

42 

479,223 

30,199 

449,024 

1,487 

SP07  SP07 

28 

319,482 

17,850 

301,631 

1,690 

SP08  SP08 

6 

63,896 

16.704 

47,192 

283 

SP09  SP09 

17 

191,689 

26.943 

164,746 

611 

SP10SP10 

8 

89,455 

46,423 

43,032 

93 

Total  Not  defined 

110 

1.245,979 

169,577 

1,076,402 

635 

TOTAL 

220 

2  500  000 

1  097  041 

_ 1.402,959 _ 

128 

Figure  7.  GaussSoft  Radial  Viewer  Sample  Report 
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